Writing, page 11

Cyber Agents Score 93% on Cybench. Nobody Put a Defender in the Benchmark.

18 minute read

· Security & privacy

In one year frontier models went from solving 15% of Cybench to 93%, and the benchmark is now described as saturated. So I went looking for an evaluation that put a defender in the environment. There is one. GPT-4 goes from winning every episode to winning half of them, and the experiment was published in 2024. Every headline capability number in this field was measured in a room with nothing fighting back.

13 Million Lines of Lean in 11 Days. The Hard Part Was Never the Proof.

14 minute read

· Science & medicine

Anthropic published a machine-checked formalization of Fermat’s Last Theorem this morning. The number everyone will quote is 13 million lines of Lean. The line that actually matters is the one saying a comparator confirmed the theorem’s statement matched mathlib’s β€” because a proof checker verifies a proof against a statement, and has nothing whatsoever to say about whether the statement means what you wanted.

17,000 Edits to a Dead German Wiki. β€˜Hijacked’ Is Doing a Lot of Work.

13 minute read

· AI safety & governance

This week’s story about AI agents writing to a German wiki gets described with four words β€” hijacked, breakout, rogue, collusion. Those are four different technical claims, and the evidence behind them runs from solid to a single anecdote. The agents did not break into anything; the wiki was already open. What leaked was the sandbox on the other side.

Astra on ARC-AGI-3: The Number That Matters Is the Gap Between the Two Scores

15 minute read

· Benchmarks & measurement

The ARC Prize Foundation reports two scores for OpenAI’s GPT-6 Astra on ARC-AGI-3: 62.7% under its provider-neutral harness and 99.9% when OpenAI’s own context management is switched on. Same model, same 55 hidden games, 37 points apart. That gap, and the fact that Astra now uses fewer actions than the median human on 96% of levels, tells you more about where frontier AI is than either headline number does.

Proposing a Material Is Easy. Knowing What You Made Is Not.

18 minute read

· Science & medicine

In January 2026, Nature published an Author Correction to the A-Lab paper. The robot chemist’s headline of 41 new compounds became 36 confirmed, 4 inconclusive, and 1 removed for having been in the training data, and the novelty claim was withdrawn outright. The correction is a better paper than the original, and it points at the real bottleneck in machine-driven materials discovery: not proposing candidates, not even making them, but establishing what came out of the furnace.

Synthetic Data Doesn’t Poison Models. Throwing Away the Real Data Does.

20 minute read

· Models, inference & pricing

Model collapse is real, reproducible, and provable in the limit β€” but every canonical demonstration of it replaces the human data with machine output. Change that one assumption and the theory changes with it: accumulate real and synthetic data and the test error is provably bounded; filter synthetic data through a verifier and it beats human data outright. What degrades in every regime, including the successful ones, is diversity.

Robot Foundation Models Generalize. The Experiments Can’t Tell You How Much.

13 minute read

· Benchmarks & measurement

Vision-language-action models really do transfer web knowledge into manipulation, and the headline gains are real. But the absolute success rates on the hard generalization splits sit between 35% and 75%, cross-embodiment data has been measured hurting as often as helping, and the standard evaluation β€” three to twenty real-world trials per condition β€” produces confidence intervals wider than the improvements being claimed.

The AI Weather Models Won. Look Closely at What They Won At.

17 minute read

· Science & medicine

Data-driven forecast models now beat operational numerical weather prediction on the headline scores, and two of them are running operationally at ECMWF. But the metric they win on pays a premium for blur, the models are initialized by the very system they beat, and the ECMWF scorecard shows the winner changing depending on whether you verify against analyses or against actual weather stations.

Why Do AI Agents Forget? The Memory Stack, and the Three Ways It Breaks

13 minute read

· Agents

Agent memory is not one thing. It is four stores and a retrieval policy, and the binding constraint is almost never capacity β€” it is that long context degrades well below its stated limit, that nearly every published memory benchmark result is self-reported by the system’s own authors, and that a persistent store is a writable attack surface.

Your Biggest Decisions Are Hiding in Unstructured Data β€” Can AI Actually Read Them Out?

7 minute read

· Analytics & decision science

The statistic that launched a thousand data platforms β€” β€˜80% of enterprise data is unstructured’ β€” turns out to be a hedged guess from a 1998 analyst note. The real story is better: AI can now read documents at human level, still struggles to extract from them faithfully, and the honest evidence on what that unlocks for decisions.

page 11 of 12 · 178 essays