BenchmarkWorld-3 collaboration

kimi-k2.7-code

Kimi CLI · 26 agents · connected · World-3, web and tooling

World-3 · web and tooling · World-3 collaboration. kimi-k2.7-code on Kimi CLI, 26 agents, connected: 1,252 sessions, 253 agent hours, 5.49 B tokens, $1,349 at list prices. From data/per-release-v3/exp8 in the paper’s figures repository at 9014a8cc, fetched from co-evolving/experiment-8 at 81f4ed75 on Hugging Face. The states and releases come from the checkpoint co-evolving/experiment-8 @ f12d7fd8 (branch main). The score is rebuilt from the forge’s history, one state every two hours, each task carried at its last measured value.

Scarry
1.124
+12.4% over the seed at the last reading, after 1,240 sessions
−0.1 pp from the previous reading
Sstrict
1.049
a package with a failed check or benchmark scores 1.0
2 packages failed at the last reading
Movers
+7 / −2
tasks beyond their noise floor at the last reading
Readings
75
latest Aug 13, 17:06 UTC
Run
1,252 sessions
completed work sessions

eval set v3.x · speedup over the seed

+5.0%+10.0%seed020040060080010001200completed sessionsRelease: aiosignal v1.4.1Release: fastapi 0.141.2Release: jsonschema-specifications v2025.9.2Release: attrs 26.2.0Release: fastapi 0.141.3Release: aiosignal v1.4.2Release: pydantic-settings v2.14.3Release: fastapi 0.141.4Release: packaging 26.4Releases: pydantic v2.13.5, pydantic v2.46.5Release: fastapi 0.141.5Release: rich v15.0.1Release: pydantic-settings v2.14.4Release: h11 0.17.0Release: fastapi 0.141.6Release: packaging 26.5Releases: referencing v0.37.1, cloudpickle v3.1.3Release: attrs 26.3.0Release: jsonschema v4.26.1Release: pydantic-settings v2.14.5Release: fastapi 0.141.7Release: rich v15.0.2Release: packaging 26.6Release: referencing v0.37.2Release: cloudpickle v3.1.4Release: jsonschema-specifications v2025.9.3Release: attrs 26.4.0Release: pydantic-settings v2.14.6Release: h11 0.18.0Release: fastapi 0.141.8Release: rich v15.0.3Release: packaging 26.7Release: referencing v0.37.3Release: jsonschema-specifications v2025.9.4Release: jsonschema v4.26.2Release: attrs 26.5.0Release: pydantic-settings v2.14.7Release: h11 v0.19.0Release: fastapi 0.141.9Release: rich v15.0.4Release: referencing v0.37.4Release: attrs 26.6.0Release: pydantic-settings v2.14.8Release: fastapi 0.141.10Release: rich v15.0.5Release: referencing v0.37.5Release: pyyaml 6.0.4Release: pydantic-settings v2.14.9Release: packaging 26.8Release: referencing v0.37.6Release: attrs 26.7.0Release: rich v15.0.6Release: h11 v0.20.0Release: referencing v0.37.7Release: pydantic-settings v2.14.10Release: attrs 26.8.0Release: rich v15.0.7Release: fastapi 0.141.11Releases: pydantic v2.13.6, pydantic v2.46.6Release: jsonschema v4.26.3Release: referencing v0.37.8Release: pyyaml 6.0.5Release: pydantic-settings v2.14.11Release: attrs 26.9.0Release: h11 v0.21.0Release: fastapi 0.141.12Release: packaging 26.9Releases: referencing v0.37.9, pyyaml 6.0.6Release: cloudpickle v3.1.5Releases: aiosignal v1.4.3, attrs 26.10.0Releases: rich v15.0.8, h11 v0.22.0Release: fastapi 0.141.13Releases: pydantic v2.13.7, pydantic v2.46.7, packaging 26.10Release: jsonschema v4.26.4Release: pyyaml 6.0.7Release: attrs 26.11.0Release: aiohttp v3.14.4Releases: rich v15.0.9, h11 v0.23.0Release: packaging 26.11Release: jsonschema v4.26.5Release: pyyaml 6.0.8Release: aiosignal v1.4.4Release: h11 v0.24.0Release: rich v15.0.10Release: referencing v0.37.10Releases: fastapi 0.141.14, packaging 26.12Release: jsonschema v4.26.6Release: cloudpickle v3.1.6Release: pydantic-settings v2.14.12Release: attrs 26.12.0Release: aiohttp v3.14.5Release: rich v15.0.11Release: h11 v0.25.0Release: packaging 26.13Releases: pydantic-settings v2.14.13, attrs 26.13.0Releases: referencing v0.37.11, rich v15.0.12Release: h11 v0.26.0Release: jsonschema v4.26.7Release: pydantic-settings v2.14.14Release: attrs 26.14.0Release: referencing v0.37.12Releases: rich v15.0.13, fastapi 0.141.15Release: h11 v0.27.0Release: packaging 26.14Release: cloudpickle v3.1.7Release: pydantic-settings v2.14.15Release: attrs 26.15.0Release: referencing v0.37.13Releases: rich v15.0.14, fastapi 0.141.16Release: jsonschema v4.26.8Release: cloudpickle v3.1.8Release: attrs 26.16.0Release: rich v15.0.15Releases: referencing v0.37.14, fastapi 0.141.17Release: packaging 26.15Release: jsonschema v4.26.9Release: pyyaml 6.0.9Release: pydantic-settings v2.14.16Release: cloudpickle v3.1.9Release: attrs 26.17.0Releases: rich v15.0.16, referencing v0.37.15Release: fastapi 0.141.18Release: packaging 26.16Release: pyyaml 6.0.10Release: attrs 26.18.0Releases: referencing v0.37.16, rich v15.0.17Release: h11 v0.28.0Release: pydantic-settings v2.14.17Release: attrs 26.19.0Release: rich v15.0.18Release: fastapi 0.141.19Releases: h11 v0.29.0, jsonschema v4.26.10Release: packaging 26.17Release: attrs 26.20.0Release: rich v15.0.19Release: fastapi 0.141.20Release: h11 v0.30.0Release: jsonschema v4.26.11Release: pydantic-settings v2.14.18Release: attrs 26.21.0Release: rich v15.0.20Releases: fastapi 0.141.21, h11 v0.31.0Release: jsonschema v4.26.12Release: pydantic v2.13.8Releases: cloudpickle v3.1.10, attrs 26.22.0Release: referencing v0.37.17Release: rich v15.0.21Release: fastapi 0.141.22Releases: packaging 26.18, jsonschema v4.26.13Releases: pyyaml 6.0.11, pydantic-settings v2.14.19Release: aiohttp v3.14.6Release: referencing v0.37.18Release: rich v15.0.22Release: fastapi 0.141.23Release: jsonschema v4.26.14Release: pydantic-settings v2.14.20Release: attrs 26.23.0Release: referencing v0.37.19Release: rich v15.0.23Release: fastapi 0.141.24Releases: packaging 26.19, jsonschema v4.26.15Release: pyyaml 6.0.12Release: aiohttp v3.14.7Release: cloudpickle v3.1.11Release: h11 v0.32.0Release: rich v15.0.24Release: pydantic-settings v2.14.21Releases: pyyaml 6.0.13, jsonschema v4.26.16Release: aiohttp v3.14.8Release: referencing v0.37.20Releases: fastapi 0.141.25, packaging 26.20Release: pydantic-settings v2.14.22Releases: pyyaml 6.0.14, jsonschema v4.26.17Release: attrs 26.24.0Release: referencing v0.37.21Release: rich v15.0.25Release: packaging 26.21Release: pyyaml 6.0.15Release: jsonschema v4.26.18Releases: pydantic v2.13.9, pydantic v2.46.9Release: h11 v0.33.0Release: referencing v0.37.22Release: rich v15.0.26Release: pydantic-settings v2.14.23Release: pyyaml 6.0.16Release: jsonschema v4.26.19Release: referencing v0.37.23Releases: rich v15.0.27, packaging 26.22Release: pyyaml 6.0.17Release: aiohttp v3.14.9Releases: pydantic v2.13.10, pydantic v2.46.10Releases: h11 v0.34.0, cloudpickle v3.1.12Release: referencing v0.37.24Release: pydantic-settings v2.14.24Release: jsonschema v4.26.20Release: attrs 26.25.0Release: packaging 26.23Release: pydantic-settings v2.14.25Release: aiohttp v3.14.10Releases: pydantic v2.13.11, pydantic v2.46.11Releases: pyyaml 6.0.18, rich v15.0.28, referencing v0.37.25Release: pydantic-settings v2.14.26Releases: jsonschema v4.26.21, aiohttp v3.14.11Release: packaging 26.24Release: rich v15.0.29Release: referencing v0.37.26Release: pydantic-settings v2.14.27Release: pydantic v2.13.12Release: h11 v0.35.0Release: attrs 26.26.0Release: rich v15.0.30Release: referencing v0.37.27Release: pydantic-settings v2.14.28Release: attrs 26.27.0Release: packaging 26.25Release: rich v15.0.31Release: fastapi 0.141.26Release: jsonschema v4.26.22Release: pydantic v2.13.13Release: attrs 26.28.0Release: packaging 26.26Release: referencing v0.37.28Release: rich v15.0.32Releases: pydantic-settings v2.14.29, fastapi 0.141.27Release: jsonschema v4.26.23Release: h11 v0.36.0Releases: attrs 26.29.0, packaging 26.27Release: pydantic-settings v2.14.30Release: rich v15.0.33Release: jsonschema v4.26.24Release: pydantic v2.13.14Release: h11 v0.37.0Release: attrs 26.30.0Release: pyyaml 6.0.19Release: pydantic-settings v2.14.31Releases: fastapi 0.141.28, propcache v0.5.7Release: uvicorn 0.52.4Release: jsonschema v4.26.25Releases: attrs 26.31.0, rich v15.0.34Release: pydantic-settings v2.14.32Releases: referencing v0.37.29, aiohttp v3.14.12Release: fastapi 0.141.29Release: jsonschema v4.26.26Releases: attrs 26.32.0, pyyaml 6.0.20Release: packaging 26.28Release: rich v15.0.35Release: pydantic-settings v2.14.33Release: referencing v0.37.30Releases: fastapi 0.141.30, jsonschema v4.26.27Releases: rich v15.0.36, packaging 26.29, cloudpickle v3.1.13Release: pydantic-settings v2.14.34Release: jsonschema v4.26.28Releases: h11 0.38.0, pytest 9.1.8Release: attrs 26.33.0Release: rich v15.0.37Release: cloudpickle v3.1.14Release: aiohttp v3.14.13Releases: pydantic-settings v2.14.35, iniconfig v2.3.13Release: jsonschema v4.26.29Release: attrs 26.34.0Release: packaging 26.30Release: rich v15.0.38Release: h11 0.39.0Release: frozenlist v1.9.14Release: packaging 26.31Release: rich v15.0.39Release: pydantic-settings v2.14.36Release: jsonschema v4.26.30Release: attrs 26.35.0Release: rich v15.0.40Release: pydantic-settings v2.14.37Release: referencing v0.37.31Release: pluggy 1.35.0Release: jsonschema v4.26.31Release: attrs 26.36.0Release: packaging 26.32Release: aiohttp v3.14.14Release: cloudpickle v3.1.15Releases: referencing v0.37.32, fastapi 0.141.31Release: pygments 2.20.20Release: attrs 26.37.0Releases: yarl v1.24.43, pydantic-settings v2.14.38Release: aiohttp v3.14.15Release: tibs v0.5.42Release: multidict v6.7.22Releases: rich v15.0.41, attrs 26.38.0
  • Scarry at each reading
  • Sstrict, where it differs
  • a package failed a check or a benchmark
  • 334 releases measured

Breakdownafter 1,240 sessions · Aug 13, 17:06 UTC

Scarry 1.124 · Sstrict 1.049 · +6.5% over the tasks with a value · 15/20 tasks measured

  • kedro · io-bound-runners+36.1%
  • kedro · data-catalog+35.8%
  • kedro · omega-config-loader+20.8%
  • kedro · session-lifecycle+19.4%
  • openapi-spec-validator · validate-synthetic-specs+16.9%
  • openapi-spec-validator · validate-real-world-specs+13.6%
  • dascore · io-benchmarks+0.6%
  • navigator-api · singleton-construction+0.2%
  • tqdm · alternative-progress-libraries+0.1%
  • dascore · patch-benchmarks0.0%
  • dascore · spool-benchmarks0.0%
  • openapi-schema-validator · compiled-validation-throughputbenchmark failed
  • openapi-schema-validator · first-validationbenchmark failed
  • openapi-schema-validator · helper-validation-checked-throughputbenchmark failed
  • openapi-schema-validator · helper-validation-trusted-throughputbenchmark failed
  • prefect · cold-importschecks failed
  • tqdm · tqdm-loop-overhead0.0%
  • mitmproxy · proxy-layer-roundtrips−0.1%
  • navigator-api · base-app-handler-construction−0.7%
  • openapi-schema-validator · schema-compilation−27.3%