Benchmarks & measurement, page 2

OpenAI Says Its Median Researcher Spends $600 a Day on Agents. That Is a Bill, Not a Result.

19 minute read

· Benchmarks & measurement

OpenAI published internal numbers on how much its researchers use coding agents, and declared it had reached its goal of an automated research intern. Almost every figure disclosed is money spent or effort deployed, and the post’s own appendix says those figures are hard to interpret. The independent evidence is thin, contradictory, and includes one randomised trial where developers were measurably slower while believing they were faster.

Astra on ARC-AGI-3: The Number That Matters Is the Gap Between the Two Scores

15 minute read

· Benchmarks & measurement

The ARC Prize Foundation reports two scores for OpenAI’s GPT-6 Astra on ARC-AGI-3: 62.7% under its provider-neutral harness and 99.9% when OpenAI’s own context management is switched on. Same model, same 55 hidden games, 37 points apart. That gap, and the fact that Astra now uses fewer actions than the median human on 96% of levels, tells you more about where frontier AI is than either headline number does.

Robot Foundation Models Generalize. The Experiments Can’t Tell You How Much.

13 minute read

· Benchmarks & measurement

Vision-language-action models really do transfer web knowledge into manipulation, and the headline gains are real. But the absolute success rates on the hard generalization splits sit between 35% and 75%, cross-embodiment data has been measured hurting as often as helping, and the standard evaluation β€” three to twenty real-world trials per condition β€” produces confidence intervals wider than the improvements being claimed.

page 2 of 2 · 20 essays on Benchmarks & measurement