In one year frontier models went from solving 15% of Cybench to 93%, and the benchmark is now described as saturated. So I went looking for an evaluation that put a defender in the environment. There is one. GPT-4 goes from winning every episode to winning half of them, and the experiment was published in 2024. Every headline capability number in this field was measured in a room with nothing fighting back.
Anthropic published a machine-checked formalization of Fermatβs Last Theorem this morning. The number everyone will quote is 13 million lines of Lean. The line that actually matters is the one saying a comparator confirmed the theoremβs statement matched mathlibβs β because a proof checker verifies a proof against a statement, and has nothing whatsoever to say about whether the statement means what you wanted.
This weekβs story about AI agents writing to a German wiki gets described with four words β hijacked, breakout, rogue, collusion. Those are four different technical claims, and the evidence behind them runs from solid to a single anecdote. The agents did not break into anything; the wiki was already open. What leaked was the sandbox on the other side.
In 2024 two analyses looked at the same Waymo crash reports and reached opposite conclusions: one found the robotaxi crashing about four times more often than a human driver, the other found large reductions. Neither side made an arithmetic error. They chose different denominators, and in per-mile crash statistics the denominator is most of the answer.
The ARC Prize Foundation reports two scores for OpenAIβs GPT-6 Astra on ARC-AGI-3: 62.7% under its provider-neutral harness and 99.9% when OpenAIβs own context management is switched on. Same model, same 55 hidden games, 37 points apart. That gap, and the fact that Astra now uses fewer actions than the median human on 96% of levels, tells you more about where frontier AI is than either headline number does.
In January 2026, Nature published an Author Correction to the A-Lab paper. The robot chemistβs headline of 41 new compounds became 36 confirmed, 4 inconclusive, and 1 removed for having been in the training data, and the novelty claim was withdrawn outright. The correction is a better paper than the original, and it points at the real bottleneck in machine-driven materials discovery: not proposing candidates, not even making them, but establishing what came out of the furnace.
Model collapse is real, reproducible, and provable in the limit β but every canonical demonstration of it replaces the human data with machine output. Change that one assumption and the theory changes with it: accumulate real and synthetic data and the test error is provably bounded; filter synthetic data through a verifier and it beats human data outright. What degrades in every regime, including the successful ones, is diversity.
Vision-language-action models really do transfer web knowledge into manipulation, and the headline gains are real. But the absolute success rates on the hard generalization splits sit between 35% and 75%, cross-embodiment data has been measured hurting as often as helping, and the standard evaluation β three to twenty real-world trials per condition β produces confidence intervals wider than the improvements being claimed.
Data-driven forecast models now beat operational numerical weather prediction on the headline scores, and two of them are running operationally at ECMWF. But the metric they win on pays a premium for blur, the models are initialized by the very system they beat, and the ECMWF scorecard shows the winner changing depending on whether you verify against analyses or against actual weather stations.
On-device inference is not compute-limited β it is memory-bandwidth-limited, and the NPU spec sheet measures the wrong thing. The accuracy curve is flat to 8-bit, gentle at 4-bit weights, and a cliff below that. And at identical 4-bit precision, the choice of runtime changes energy per token by tenfold.
Agent memory is not one thing. It is four stores and a retrieval policy, and the binding constraint is almost never capacity β it is that long context degrades well below its stated limit, that nearly every published memory benchmark result is self-reported by the systemβs own authors, and that a persistent store is a writable attack surface.
EY says 98% of AI investors see positive ROI. McKinseyβs survey says 63% see no earnings impact at all. Bain calls the spending a bet on returns that havenβt arrived. They can all be right, because surveys measure belief β and the experiments, which measure work, tell a much more consistent story.
There is no thermometer for general intelligence. There are three families of instrument β difficulty, duration, and economic work β each with a documented failure mode, and at least two of them have measurement error large enough to swallow a year of apparent progress.
The statistic that launched a thousand data platforms β β80% of enterprise data is unstructuredβ β turns out to be a hedged guess from a 1998 analyst note. The real story is better: AI can now read documents at human level, still struggles to extract from them faithfully, and the honest evidence on what that unlocks for decisions.
Automated traffic passed human traffic on the web this year, the payment networks spent 2025 building rails for machine buyers, and the research on serving agents as customers is three decades old. What changes for customer success when the user of record canβt attend the QBR.