OpenAI published internal numbers on how much its researchers use coding agents, and declared it had reached its goal of an automated research intern. Almost every figure disclosed is money spent or effort deployed, and the postβs own appendix says those figures are hard to interpret. The independent evidence is thin, contradictory, and includes one randomised trial where developers were measurably slower while believing they were faster.
In 2024 two analyses looked at the same Waymo crash reports and reached opposite conclusions: one found the robotaxi crashing about four times more often than a human driver, the other found large reductions. Neither side made an arithmetic error. They chose different denominators, and in per-mile crash statistics the denominator is most of the answer.
The ARC Prize Foundation reports two scores for OpenAIβs GPT-6 Astra on ARC-AGI-3: 62.7% under its provider-neutral harness and 99.9% when OpenAIβs own context management is switched on. Same model, same 55 hidden games, 37 points apart. That gap, and the fact that Astra now uses fewer actions than the median human on 96% of levels, tells you more about where frontier AI is than either headline number does.
Vision-language-action models really do transfer web knowledge into manipulation, and the headline gains are real. But the absolute success rates on the hard generalization splits sit between 35% and 75%, cross-embodiment data has been measured hurting as often as helping, and the standard evaluation β three to twenty real-world trials per condition β produces confidence intervals wider than the improvements being claimed.
There is no thermometer for general intelligence. There are three families of instrument β difficulty, duration, and economic work β each with a documented failure mode, and at least two of them have measurement error large enough to swallow a year of apparent progress.
page 2 of 2 · 20 essays on Benchmarks & measurement