BenchmarkWorld-3 collaboration

gpt-5.6-sol

Codex · 26 agents · connected · World-3, web and tooling

World-3 · web and tooling · World-3 collaboration. gpt-5.6-sol on Codex, 26 agents, connected: 877 sessions, 113 agent hours, 2.18 B tokens, $1,795 at list prices. From data/per-release-v3/exp6 in the paper’s figures repository at 9014a8cc, fetched from co-evolving/experiment-6 at 9344a95b on Hugging Face. The states and releases come from the checkpoint co-evolving/experiment-6 @ 3f87dd17 (branch main). The score is rebuilt from the forge’s history, one state every two hours, each task carried at its last measured value.

Scarry
1.060
+6.0% over the seed at the last reading, after 799 sessions
+0.5 pp from the previous reading
Sstrict
1.060
a package with a failed check or benchmark scores 1.0
Movers
+13 / −0
tasks beyond their noise floor at the last reading
Readings
10
latest Aug 5, 14:49 UTC
Run
877 sessions
completed work sessions

eval set v3.x · speedup over the seed

+2.0%+4.0%+6.0%seed0200400600800completed sessionsRelease: rich v15.0.1Release: fastapi 0.141.2Release: packaging 26.4Release: jsonschema v4.26.1Release: cloudpickle v3.1.3Releases: referencing v0.37.1, pydantic-settings v2.14.3Releases: jsonschema-specifications v2026.8.1, h11 v0.16.1Releases: pyyaml 6.0.4, jsonschema v4.26.2Releases: attrs 26.2.0, rich v15.0.2Release: fastapi 0.141.3Release: packaging 26.5Release: jsonschema-specifications v2026.8.2Releases: referencing v0.37.2, pydantic-settings v2.14.4Release: h11 v0.16.2Release: jsonschema-specifications v2026.8.3Release: pydantic v2.13.5Release: packaging 26.6Release: aiosignal v1.4.1Release: jsonschema v4.26.3Releases: rich v15.0.3, attrs 26.3.0Release: referencing v0.37.3Release: fastapi 0.141.4Release: pydantic-settings v2.14.5Release: h11 v0.16.3Release: cloudpickle v3.1.4Release: jsonschema-specifications v2026.8.4Release: pydantic v2.13.6Release: fastapi 0.141.5Release: rich v15.0.4Releases: jsonschema v4.26.4, h11 v0.16.4, packaging 26.7, pydantic-settings v2.14.6Release: pyyaml 6.0.5Release: referencing v0.37.4Release: fastapi 0.141.6Release: pydantic v2.13.7Releases: cloudpickle v3.1.5, propcache v0.5.6Release: jsonschema v4.26.5Release: referencing v0.37.5Release: h11 v0.16.5Releases: rich v15.0.5, aiosignal v1.4.2Release: jsonschema v4.26.6Release: pydantic v2.13.8Release: packaging 26.8Release: fastapi 0.141.7Release: pydantic-settings v2.14.7Release: rich v15.0.6Releases: pyyaml 6.0.6, attrs 26.4.0Release: pydantic v2.13.9Release: fastapi 0.141.8Release: pydantic-settings v2.14.8Release: cloudpickle v3.1.6Release: referencing v0.37.6Release: rich v15.0.7Release: jsonschema v4.26.7Release: packaging 26.9Release: jsonschema-specifications v2026.8.5Release: pydantic v2.13.10Release: pydantic-settings v2.14.9Release: yarl v1.24.7Release: referencing v0.37.7Releases: rich v15.0.8, jsonschema v4.26.8Release: attrs 26.5.0Release: h11 v0.16.6Release: pydantic v2.13.11Release: fastapi 0.141.9Releases: pydantic-settings v2.14.10, referencing v0.37.8Release: rich v15.0.9Release: packaging 26.10Release: jsonschema v4.26.9Release: fastapi 0.141.10Releases: h11 v0.16.7, referencing v0.37.9Release: jsonschema v4.26.10Release: jsonschema v4.26.11Release: cloudpickle v3.1.7Release: rich v15.0.10Release: aiosignal v1.4.3Release: referencing v0.37.10Release: pydantic-settings v2.14.11Release: fastapi 0.141.11Release: attrs 26.6.0Release: jsonschema v4.26.12Release: packaging 26.11Release: cloudpickle v3.1.8Release: jsonschema v4.26.13Release: fastapi 0.141.12Release: referencing v0.37.11Release: iniconfig v2.3.9Release: jsonschema v4.26.14Releases: cloudpickle v3.1.9, pydantic v2.13.12Release: h11 v0.16.8Release: rich v15.0.11Release: attrs 26.7.0Release: jsonschema v4.26.15Release: aiosignal v1.4.4Release: pydantic-settings v2.14.12Release: referencing v0.37.12Release: jsonschema v4.26.16Release: uvicorn 0.51.4Release: pydantic v2.13.13Release: pytest 9.1.7Release: jsonschema v4.26.17Release: cloudpickle v3.1.10Release: referencing v0.37.13Release: pyyaml 6.0.7Releases: aiosignal v1.4.5, jsonschema v4.26.18Release: fastapi 0.141.13Release: rich v15.0.12Release: packaging 26.12Release: cloudpickle v3.1.11Releases: referencing v0.37.14, pydantic v2.13.14, pydantic-settings v2.14.13Release: jsonschema v4.26.19Release: attrs 26.8.0Release: fastapi 0.141.14Release: tibs v0.5.24Release: pluggy 1.6.6Release: pygments 2.20.8Release: rich v15.0.13Release: jsonschema v4.26.20Release: referencing v0.37.15Release: packaging 26.13Releases: frozenlist v1.8.12, cloudpickle v3.1.12Release: pydantic-settings v2.14.14Release: multidict v6.7.12Release: jsonschema v4.26.21
  • Scarry at each reading
  • 143 releases measured

Breakdownafter 799 sessions · Aug 5, 14:49 UTC

Scarry 1.060 · Sstrict 1.060 · +6.0% over the tasks with a value · 20/20 tasks measured

  • openapi-schema-validator · schema-compilation+26.1%
  • openapi-schema-validator · compiled-validation-throughput+17.6%
  • openapi-schema-validator · first-validation+16.8%
  • kedro · omega-config-loader+14.3%
  • openapi-schema-validator · helper-validation-checked-throughput+9.6%
  • openapi-schema-validator · helper-validation-trusted-throughput+9.1%
  • prefect · cold-imports+6.8%
  • kedro · io-bound-runners+5.6%
  • kedro · data-catalog+5.4%
  • kedro · session-lifecycle+5.3%
  • openapi-spec-validator · validate-synthetic-specs+4.8%
  • openapi-spec-validator · validate-real-world-specs+2.8%
  • dascore · io-benchmarks+0.7%
  • tqdm · alternative-progress-libraries0.0%
  • dascore · patch-benchmarks0.0%
  • tqdm · tqdm-loop-overhead0.0%
  • mitmproxy · proxy-layer-roundtrips0.0%
  • navigator-api · singleton-construction−0.1%
  • navigator-api · base-app-handler-construction−0.2%
  • dascore · spool-benchmarks−0.3%