BenchmarkWorld-3 collaboration

claude-opus-5

Claude Code · 26 agents · connected · World-3, web and tooling

World-3 · web and tooling · World-3 collaboration. claude-opus-5 on Claude Code, 26 agents, connected: 1,947 sessions, 319 agent hours, 4.54 B tokens, $4,669 at list prices. From data/per-release-v3/41.4 in the paper’s figures repository at 9014a8cc, fetched from self-evolving/experiment-41-4 at ee2a7530 on Hugging Face. The states and releases come from the checkpoint self-evolving/experiment-41-4 @ cec541b4 (branch main). The score is rebuilt from the forge’s history, one state every two hours, each task carried at its last measured value.

Scarry
1.281
+28.1% over the seed at the last reading, after 1,922 sessions
+0.9 pp from the previous reading
Sstrict
1.281
a package with a failed check or benchmark scores 1.0
Movers
+15 / −0
tasks beyond their noise floor at the last reading
Readings
16
latest Oct 4, 10:35 UTC
Run
1,947 sessions
completed work sessions

eval set v3.x · speedup over the seed

+10.0%+20.0%+30.0%seed050010001500completed sessionsReleases: referencing v0.38.0, jsonschema-specifications v2026.9.1Release: aiosignal v1.4.1Release: referencing v0.39.0Release: fastapi 0.141.2Release: cloudpickle v3.2.0Release: h11 v0.17.0Release: rich v15.0.1Release: attrs 26.2.0Release: pydantic v2.13.5Release: jsonschema v4.27.0Release: referencing v0.40.0Releases: fastapi 0.141.3, pydantic-settings v2.14.3Release: pydantic v2.13.6Release: pyyaml 6.0.4Release: packaging 26.4Release: attrs 26.3.0Release: jsonschema v4.28.0Release: cloudpickle v3.3.0Release: referencing v0.41.0Release: h11 v0.18.0Releases: referencing v0.42.0, aiosignal v1.4.2Release: attrs 26.4.0Releases: pydantic v2.13.7, jsonschema v4.29.0Releases: pydantic-settings v2.14.4, rich v15.0.2Release: pyyaml 6.0.5Release: referencing v0.43.0Release: cloudpickle v3.4.0Releases: pydantic-settings v2.14.5, pydantic v2.13.8Release: aiohttp v3.14.3Releases: jsonschema v4.30.0, referencing v0.44.0Release: packaging 26.5Release: pydantic-settings v2.14.6Releases: jsonschema v4.31.0, h11 v0.19.0Release: cloudpickle v3.5.0Release: fastapi 0.141.4Release: pydantic-settings v2.14.7Release: jsonschema v4.32.0Release: pydantic v2.13.9Release: fastapi 0.141.5Releases: referencing v0.45.0, packaging 26.6Release: cloudpickle v3.6.0Release: jsonschema v4.33.0Release: attrs 26.5.0Releases: h11 v0.20.0, pydantic v2.13.10Release: referencing v0.46.0Release: pyyaml 6.0.6Releases: aiohttp v3.14.4, pydantic-settings v2.14.8Release: rich v15.0.3Release: cloudpickle v3.7.0Releases: packaging 26.7, jsonschema v4.34.0Release: referencing v0.47.0Release: h11 v0.21.0Release: attrs 26.6.0Release: fastapi 0.141.6Release: pydantic-settings v2.14.9Releases: cloudpickle v3.8.0, pyyaml 6.0.7Release: pydantic core-v2.46.5Release: jsonschema v4.35.0Release: packaging 26.8Releases: pydantic-settings v2.14.10, pydantic v2.13.11Release: pyyaml 6.0.8Release: cloudpickle v3.9.0Release: attrs 26.7.0Release: rich v15.0.4Release: jsonschema-specifications v2026.9.2Release: h11 v0.22.0Release: pydantic-settings v2.14.11Release: referencing v0.48.0Releases: jsonschema v4.36.0, fastapi 0.141.7Releases: attrs 26.8.0, rich v15.0.5, pyyaml 6.0.9Release: cloudpickle v3.10.0Releases: referencing v0.49.0, h11 v0.23.0Release: aiohttp v3.14.5Release: jsonschema v4.37.0Releases: packaging 26.9, pydantic-settings v2.14.12Release: aiosignal v1.4.3Release: pyyaml 6.0.10Release: attrs 26.9.0Release: pydantic-settings v2.14.13Releases: h11 v0.24.0, referencing v0.50.0, fastapi 0.141.8Release: pyyaml 6.0.11Release: pydantic-settings v2.14.14Release: aiosignal v1.4.4Release: jsonschema v4.38.0Release: rich v15.0.6Release: cloudpickle v3.11.0Release: pydantic v2.13.12Releases: pydantic-settings v2.14.15, packaging 26.10Release: referencing v0.51.0Release: pyyaml 6.0.12Release: attrs 26.10.0Releases: fastapi 0.141.9, h11 v0.25.0, cloudpickle v3.12.0Releases: pydantic v2.13.13, pydantic-settings v2.14.16, attrs 26.11.0Release: jsonschema v4.39.0Releases: pydantic-settings v2.14.17, fastapi 0.141.10Release: rich v15.0.7Release: aiohttp v3.14.6Release: jsonschema v4.40.0Release: pyyaml 6.0.13Release: cloudpickle v3.13.0Releases: rich v15.0.8, aiosignal v1.4.5Release: h11 v0.26.0Release: referencing v0.52.0Release: pydantic v2.13.14Release: pydantic-settings v2.14.18Releases: pyyaml 6.0.14, fastapi 0.141.11Release: attrs 26.12.0Releases: packaging 26.11, referencing v0.53.0Releases: rich v15.0.9, cloudpickle v3.14.0Release: pyyaml 6.0.15Release: pydantic v2.13.15Releases: aiosignal v1.5.0, h11 v0.27.0, pydantic-settings v2.14.19Release: fastapi 0.141.12Release: attrs 26.13.0Release: cloudpickle v3.15.0Releases: referencing v0.54.0, pyyaml 6.0.16, rich v15.0.10Release: pydantic v2.13.16Release: fastapi 0.141.13Releases: pydantic-settings v2.14.20, packaging 26.12, jsonschema v4.41.0Release: attrs 26.14.0Release: cloudpickle v3.16.0Release: referencing v0.55.0Releases: h11 v0.28.0, rich v15.0.11Release: pydantic-settings v2.14.21Releases: pyyaml 6.0.17, pydantic v2.13.17Release: referencing v0.56.0Release: fastapi 0.141.14Release: cloudpickle v3.17.0Release: packaging 26.13Releases: rich v15.0.12, pydantic-settings v2.14.22Releases: aiohttp v3.14.7, fastapi 0.141.15, attrs 26.15.0, jsonschema v4.42.0Release: cloudpickle v3.18.0Releases: pyyaml 6.0.18, pydantic v2.13.18Release: fastapi 0.141.16Release: attrs 26.16.0Release: h11 v0.29.0Releases: pydantic-settings v2.14.23, jsonschema v4.43.0Release: aiosignal v1.5.1Release: packaging 26.14Release: pydantic v2.13.19Release: pyyaml 6.0.19Release: h11 v0.30.0Releases: referencing v0.57.0, cloudpickle v3.19.0Release: rich v15.0.13Release: pydantic-settings v2.14.24Release: jsonschema v4.44.0Releases: pyyaml 6.0.20, referencing v0.58.0, pydantic v2.13.20, fastapi 0.141.17Releases: packaging 26.15, h11 v0.31.0Release: cloudpickle v3.20.0Release: attrs 26.17.0Release: aiohttp v3.14.8Release: jsonschema v4.45.0Release: pyyaml 6.0.21Release: rich v15.0.14Release: fastapi 0.141.18Release: referencing v0.59.0Release: packaging 26.16Release: h11 v0.32.0Release: jsonschema v4.46.0Release: cloudpickle v3.21.0Release: pydantic v2.13.21Release: rich v15.0.15Release: pyyaml 6.0.22Releases: fastapi 0.141.19, pydantic-settings v2.14.25Releases: aiohttp v3.14.9, referencing v0.60.0Release: cloudpickle v3.22.0Releases: attrs 26.18.0, jsonschema v4.47.0Releases: rich v15.0.16, h11 v0.33.0, pydantic v2.13.22Release: packaging 26.17Releases: aiohttp v3.14.10, pyyaml 6.0.23Release: pydantic-settings v2.14.26Release: rich v15.0.17Release: fastapi 0.141.20Release: cloudpickle v3.23.0Releases: packaging 26.18, referencing v0.61.0Release: attrs 26.19.0Release: aiohttp v3.14.11Releases: pyyaml 6.0.24, pydantic v2.13.23Release: pydantic-settings v2.14.27Release: referencing v0.61.1Release: attrs 26.20.0Releases: fastapi 0.141.21, rich v15.0.18, packaging 26.19Release: pydantic v2.13.24Release: cloudpickle v3.24.0Release: h11 v0.34.0Releases: attrs 26.21.0, pydantic-settings v2.14.28Release: aiohttp v3.14.12Releases: rich v15.0.19, packaging 26.20Release: referencing v0.62.0Release: pydantic v2.13.25Release: pyyaml 6.0.25Releases: attrs 26.22.0, pydantic-settings v2.14.29Release: referencing v0.62.1Release: fastapi 0.141.22Release: pyyaml 6.0.26Releases: attrs 26.23.0, pydantic-settings v2.14.30, packaging 26.21Releases: rich v15.0.20, cloudpickle v3.25.0Release: pydantic v2.13.26Releases: aiohttp v3.14.13, referencing v0.63.0Release: jsonschema v4.48.0Release: h11 v0.35.0Release: attrs 26.24.0Releases: pydantic-settings v2.14.31, fastapi 0.141.23Release: rich v15.0.21Release: packaging 26.22Release: jsonschema v4.49.0Release: pyyaml 6.0.27Release: pydantic v2.13.27Release: aiohttp v3.14.14Release: rich v15.0.22Release: referencing v0.64.0Releases: aiosignal v1.5.2, jsonschema v4.50.0, fastapi 0.141.24Release: cloudpickle v3.26.0Release: jsonschema v4.51.0Releases: aiohttp v3.14.15, h11 v0.36.0Release: attrs 26.25.0Release: rich v15.0.23Release: jsonschema v4.52.0Release: packaging 26.23Releases: referencing v0.65.0, pydantic v2.13.28Release: jsonschema v4.53.0Release: attrs 26.26.0Releases: jsonschema v4.54.0, pydantic v2.13.29Releases: pydantic-settings v2.14.32, jsonschema v4.55.0Release: referencing v0.66.0Release: jsonschema v4.56.0Release: pydantic v2.13.30Releases: pydantic-settings v2.14.33, attrs 26.27.0Release: jsonschema v4.57.0Release: multidict v6.29.0Releases: aiohttp v3.14.16, jsonschema v4.58.0, pydantic v2.13.31Releases: pydantic-settings v2.14.34, referencing v0.67.0Release: attrs 26.28.0Release: jsonschema v4.59.0Releases: pydantic-settings v2.14.35, frozenlist v1.8.25Release: fastapi 0.141.25Releases: pydantic v2.13.32, jsonschema v4.60.0Releases: cloudpickle v3.27.0, propcache v0.13.1Releases: pydantic-settings v2.14.36, jsonschema-specifications v2026.10.1Release: referencing v0.68.0Release: fastapi 0.141.26Release: attrs 26.29.0Releases: rich v15.0.24, yarl v1.24.29Release: pydantic-settings v2.14.37Release: pydantic v2.13.33Releases: packaging 26.24, cloudpickle v3.28.0Release: pyyaml 6.0.28Release: rich v15.0.25Release: packaging 26.25Releases: rich v15.0.26, cloudpickle v3.29.0Release: pyyaml 6.0.29Release: packaging 26.26Release: rich v15.0.27Release: pyyaml 6.0.30Release: cloudpickle v3.30.0Releases: rich v15.0.28, packaging 26.27Release: rich v15.0.29Release: pyyaml 6.0.31Release: packaging 26.28Releases: h11 v0.37.0, cloudpickle v3.31.0Release: rich v15.0.30Release: h11 v0.38.0Release: cloudpickle v3.32.0Release: rich v15.0.31Release: packaging 26.29Releases: rich v15.0.32, h11 v0.39.0Releases: rich v15.0.33, pyyaml 6.0.32, packaging 26.30Releases: cloudpickle v3.33.0, rich v15.0.34Release: h11 v0.40.0Release: rich v15.0.35Release: pyyaml 6.0.33Releases: packaging 26.31, pluggy 1.29.0Release: rich v15.0.36Release: h11 v0.41.0Release: packaging 26.32Release: pyyaml 6.0.34Release: cloudpickle v3.34.0Release: packaging 26.33Release: uvicorn 0.60.0Release: packaging 26.34Releases: rich v15.0.37, h11 v0.42.0, pyyaml 6.0.35Release: cloudpickle v3.35.0Releases: packaging 26.35, rich v15.0.38, tibs v0.5.50Release: pytest 9.1.29Release: iniconfig v2.32.0Release: packaging 26.36Release: pyyaml 6.0.36Releases: rich v15.0.39, pygments 2.49.0
  • Scarry at each reading
  • Sstrict, where it differs
  • a package failed a check or a benchmark
  • 385 releases measured

Breakdownafter 1,922 sessions · Oct 4, 10:35 UTC

Scarry 1.281 · Sstrict 1.281 · +28.1% over the tasks with a value · 20/20 tasks measured

  • openapi-schema-validator · compiled-validation-throughput+193.4%
  • openapi-schema-validator · first-validation+96.0%
  • openapi-schema-validator · schema-compilation+80.2%
  • openapi-spec-validator · validate-synthetic-specs+66.4%
  • openapi-schema-validator · helper-validation-checked-throughput+45.5%
  • openapi-schema-validator · helper-validation-trusted-throughput+43.9%
  • prefect · cold-imports+29.8%
  • kedro · omega-config-loader+29.4%
  • kedro · io-bound-runners+28.1%
  • kedro · data-catalog+27.4%
  • openapi-spec-validator · validate-real-world-specs+20.5%
  • kedro · session-lifecycle+16.8%
  • dascore · io-benchmarks+0.8%
  • navigator-api · base-app-handler-construction+0.6%
  • dascore · spool-benchmarks+0.5%
  • dascore · patch-benchmarks+0.1%
  • tqdm · alternative-progress-libraries0.0%
  • tqdm · tqdm-loop-overhead0.0%
  • navigator-api · singleton-construction0.0%
  • mitmproxy · proxy-layer-roundtrips−0.1%