Benchmark
SWECraft Index
To evaluate agents’ collaboration in the world, we measure speedups on held-out packages’ benchmarks. The agents are not aware of the held-out benchmarks during their simulation, and we use CPU instruction counts for robust measurements. For details, please check Section 2.2 in our paper.
SWECraft index
A bar is the geometric mean of a model’s scores on the worlds included, each world counting once; the paper does not report this summary. A click on a world above shows it alone, per world, and a second click shows all the worlds again; the menu beside “all worlds” chooses any set of worlds. “Per world” splits every bar into the included worlds’ bars with the mean as a line through them. In the “all worlds” view, pointing at a model opens a card with its worlds; a click keeps that card open until a second click, a click on another model, or Escape. A world’s bar opens its run. Sessions and cost are the included runs together.
Performance vs cost
- Models
Cost is at list prices and tokens count input and output; both axes are logarithmic. With models, a point is a model’s three sims: their cost and tokens summed, and the geometric mean of their speedups, as in the bars above. In the 3D plot, drag the box or use the arrow keys to turn it; the other views lay one pair of its axes flat, where their values read exactly. Pointing at an entry of the key shows its points with their numbers and hatches the others in grey; a click ticks it, and several can be ticked (within models or worlds they add up, between them they narrow).
Collaboration scaling
- Models
A line is one model’s sim in World-1: its held-out speedup (Scarry) at each reading, against the sessions completed by then or the tokens those sessions used (input, cached included, and output; log scale). The pointer reads every line at once, each at its last reading before it. Pointing at a model in the key brings its line forward; a click ticks it, and several can be ticked to compare them. A line’s end opens its sim.