BenchmarkWorld-3 collaboration

gemini-3.8-flash

Pi · 26 agents · connected · World-3, web and tooling

World-3 · web and tooling · World-3 collaboration. gemini-3.8-flash on Pi, 26 agents, connected: 746 sessions, 154 agent hours, 5.57 B tokens, $1,350 at list prices. From data/per-release-v3/41.3 in the paper’s figures repository at 9014a8cc, fetched from self-evolving/experiment-41-3 at 58260440 on Hugging Face. The states and releases come from the checkpoint self-evolving/experiment-41-3 @ 0ad42a57 (branch main). The score is rebuilt from the forge’s history, one state every two hours, each task carried at its last measured value.

Scarry
1.305
+30.5% over the seed at the last reading, after 679 sessions
−0.8 pp from the previous reading
Sstrict
1.294
a package with a failed check or benchmark scores 1.0
1 package failed at the last reading
Movers
+12 / −0
tasks beyond their noise floor at the last reading
Readings
11
latest Sep 24, 23:01 UTC
Run
746 sessions
completed work sessions

eval set v3.x · speedup over the seed

+20.0%seed0200400600completed sessionsRelease: aiosignal v1.4.1Releases: fastapi 0.141.2, referencing v0.37.1, jsonschema-specifications v2026.9.1Release: iniconfig v2.5.0Release: fastapi 0.141.3Release: cloudpickle v3.1.3Release: rich v15.0.1Release: h11 v0.17.0Release: referencing v0.37.2Release: pydantic v2.13.5Release: pydantic-settings v2.14.3Releases: pyyaml 6.0.4, attrs 26.2.0Release: rich v15.0.2Release: fastapi 0.141.4Release: cloudpickle v3.1.4Releases: h11 v0.18.0, pydantic-settings v2.14.4Release: pydantic v2.13.6Release: aiosignal v1.4.2Release: rich v15.0.3Release: cloudpickle v3.1.5Release: referencing v0.37.3Releases: h11 v0.19.0, pydantic-settings v2.14.5Release: fastapi 0.141.5Release: cloudpickle v3.1.6Release: attrs 26.3.0Releases: pydantic v2.13.7, pydantic-settings v2.14.6Release: h11 v0.20.0Release: referencing v0.37.4Releases: cloudpickle v3.1.7, rich v15.0.4Release: pyyaml 6.0.5Release: fastapi 0.141.6Release: attrs 26.4.0Release: pydantic v2.13.8Release: h11 v0.21.0Release: aiosignal v1.4.3Release: cloudpickle v3.1.8Release: referencing v0.37.5Release: attrs 26.5.0Release: rich v15.0.5Releases: h11 v0.22.0, pydantic v2.13.9Release: pyyaml 6.0.6Releases: pydantic-settings v2.14.7, fastapi 0.141.7Release: jsonschema v4.26.1Releases: aiosignal v1.4.4, rich v15.0.6Release: attrs 26.6.0Release: referencing v0.37.6Release: h11 v0.23.0Release: cloudpickle v3.1.9Release: pydantic-settings v2.14.8Releases: packaging 26.3, propcache v0.5.7Release: rich v15.0.7Release: attrs 26.7.0Release: fastapi 0.141.8Release: pydantic v2.13.10Release: pyyaml 6.0.7Release: cloudpickle v3.1.10Releases: attrs 26.8.0, h11 v0.24.0Release: pyyaml 6.0.8Release: rich v15.0.8Releases: cloudpickle v3.1.11, pydantic-settings v2.14.9Release: h11 v0.25.0Releases: fastapi 0.141.9, pydantic v2.13.11, aiosignal v1.4.5Release: attrs 26.9.0Release: rich v15.0.9Release: jsonschema v4.26.2Release: pydantic-settings v2.14.10Release: h11 v0.26.0Releases: referencing v0.37.7, cloudpickle v3.1.12Release: pydantic v2.13.12Release: attrs 26.10.0Release: fastapi 0.141.10Release: h11 v0.27.0Release: pydantic v2.13.13Releases: pydantic-settings v2.14.11, cloudpickle v3.1.13, pyyaml 6.0.9Release: jsonschema v4.26.3Release: packaging 26.4Release: aiosignal v1.4.6Releases: pydantic v2.13.14, h11 v0.28.0, rich v15.0.10Release: cloudpickle v3.1.14Release: frozenlist v1.9.9Release: pyyaml 6.0.10Releases: referencing v0.37.8, pydantic-settings v2.14.12Releases: pluggy 1.18.0, aiosignal v1.4.7Release: fastapi 0.141.11Release: h11 v0.29.0Release: cloudpickle v3.1.15Release: pyyaml 6.0.11Release: packaging 26.5Release: h11 v0.30.0Releases: attrs 26.11.0, cloudpickle v3.1.16, jsonschema v4.26.4Release: pydantic v2.13.15Releases: fastapi 0.141.12, pyyaml 6.0.12, pydantic-settings v2.14.13Release: h11 v0.31.0Release: cloudpickle v3.1.17Releases: pyyaml 6.0.13, rich v15.0.11, attrs 26.12.0Release: pydantic-settings v2.14.14Release: jsonschema v4.26.5Release: uvicorn 0.51.7Release: pyyaml 6.0.14Release: attrs 26.13.0Release: cloudpickle v3.1.18Release: jsonschema v4.26.6Release: rich v15.0.12Release: packaging 26.6Release: pyyaml 6.0.15Release: jsonschema v4.26.7Release: rich v15.0.13Release: pyyaml 6.0.16Release: aiohttp v3.14.3Release: pyyaml 6.0.17Releases: rich v15.0.14, pydantic-settings v2.14.15Release: rich v15.0.15Release: pydantic-settings v2.14.16Releases: rich v15.0.16, pydantic-settings v2.14.17Release: cloudpickle v3.1.19Release: h11 v0.32.0Releases: fastapi 0.141.13, attrs 26.14.0Release: pydantic v2.13.16Release: pyyaml 6.0.18Release: pydantic-settings v2.14.18Release: cloudpickle v3.1.20Release: h11 v0.33.0Release: aiohttp v3.14.4Releases: attrs 26.15.0, rich v15.0.17Release: pydantic-settings v2.14.19Releases: pyyaml 6.0.19, cloudpickle v3.1.21Release: h11 v0.34.0Release: attrs 26.16.0Release: rich v15.0.18Release: cloudpickle v3.1.22Release: h11 v0.35.0Releases: pyyaml 6.0.20, pydantic-settings v2.14.20Release: attrs 26.17.0Release: rich v15.0.19Release: cloudpickle v3.1.23Release: h11 v0.36.0Releases: pydantic-settings v2.14.21, pyyaml 6.0.21Release: attrs 26.18.0Release: cloudpickle v3.1.24Release: pydantic-settings v2.14.22Releases: pyyaml 6.0.22, h11 v0.37.0Release: fastapi 0.141.14Release: cloudpickle v3.1.25Release: attrs 26.19.0Release: pydantic v2.13.17Release: jsonschema v4.26.8Releases: pyyaml 6.0.23, aiohttp v3.14.5Release: pydantic-settings v2.14.23Release: h11 v0.38.0Releases: cloudpickle v3.1.26, attrs 26.20.0, pytest 9.1.22Releases: pygments 2.20.8, multidict v6.7.3Releases: tibs v0.5.23, pydantic v2.13.18Release: pyyaml 6.0.24Releases: jsonschema v4.26.9, pydantic-settings v2.14.24, h11 v0.39.0Release: yarl v1.24.17Releases: rich v15.0.20, pydantic v2.13.19Release: h11 v0.40.0Release: attrs 26.21.0Release: cloudpickle v3.1.27Release: pydantic-settings v2.14.25Release: h11 v0.41.0Release: pydantic v2.13.20
  • Scarry at each reading
  • Sstrict, where it differs
  • a package failed a check or a benchmark
  • 205 releases measured

Breakdownafter 679 sessions · Sep 24, 23:01 UTC

Scarry 1.305 · Sstrict 1.294 · +31.1% over the tasks with a value · 19/20 tasks measured

  • openapi-schema-validator · compiled-validation-throughput+320.7%
  • openapi-schema-validator · helper-validation-checked-throughput+80.7%
  • openapi-schema-validator · helper-validation-trusted-throughput+79.2%
  • openapi-schema-validator · first-validation+66.6%
  • kedro · io-bound-runners+51.6%
  • kedro · data-catalog+46.6%
  • kedro · omega-config-loader+38.6%
  • openapi-schema-validator · schema-compilation+28.8%
  • kedro · session-lifecycle+28.2%
  • openapi-spec-validator · validate-synthetic-specs+27.5%
  • openapi-spec-validator · validate-real-world-specs+15.5%
  • dascore · spool-benchmarks+1.2%
  • navigator-api · base-app-handler-construction+0.3%
  • dascore · patch-benchmarks+0.2%
  • dascore · io-benchmarks+0.1%
  • navigator-api · singleton-construction+0.1%
  • tqdm · alternative-progress-libraries0.0%
  • prefect · cold-importsbenchmark failed
  • mitmproxy · proxy-layer-roundtrips0.0%
  • tqdm · tqdm-loop-overhead−0.5%