Benchmarks & measurement

Essays that read AI benchmark claims and leaderboards back to their denominators, graders and trial counts.

The Alert Was Canceled Fifteen Minutes Before the Police Were Sent

17 minute read

· Benchmarks & measurement

Four Kenwood High School students sued Omnilert, Baltimore County Public Schools and county police this week over an October 2025 night when an AI gun detector flagged a bag of Doritos and armed officers handcuffed them. The complaint calls the system’s 99 percent false-positive rate the failure. Maryland’s inspector general measured something narrower, and the alarm-fatigue literature puts that number in familiar company. What failed that night was who could act on an alert after it had been canceled.

Zero False Positives in Twenty-Five

17 minute read

· Benchmarks & measurement

Dario Caldara and JesΓΊs FernΓ‘ndez-Villaverde ran 25 real referee reports through an AI detector, then had two chatbots proofread, edit, rewrite and replace them. The detector called every original human, and every report written from scratch recommended major revision of an already published paper. The second result is the stronger one. Twenty-five clean reports can only rule out a false-positive rate above about 11 percent, and that bound matters a great deal once a journal starts acting on flags.

Forty Percent Fewer Clicks, on Forty-One Percent of Searches

21 minute read

· Benchmarks & measurement

A preregistered field experiment by Saharsh Agarwal and Ananya Sen hid Google’s AI Overviews from randomly chosen desktop users and found outbound clicks on affected searches fell from 0.62 to 0.37 per search, a 39.8 percent cut. That is the number in the headlines. The number a publisher’s analytics will show depends on how many searches get an Overview, and the published estimates of that share run from 18 to 67 percent.

The Debate Trailed the Vote

14 minute read

· Benchmarks & measurement

A new synthesis of nine multi-agent studies finds that once the single model gets the same number of samples or the same budget, the multi-agent systems average 4.2 points worse and come out ahead in about a quarter of comparisons. In one study, five agents debating beat a single answer by 4.5 points, and five answers with no conversation at all beat it by 7.6. Researchers studying human groups hit the same baseline problem decades ago.

Twelve by Poll, 1.9 by Benchmark, 1.1 on Another Curve

17 minute read

· Benchmarks & measurement

METR spent eleven weeks inside Anthropic and concluded that capability progress there has roughly doubled since October 2025. The headline figure is a 1.9x acceleration on Anthropic’s version of the Epoch Capabilities Index. The report’s own appendix shows that one modeling choice, which the data cannot settle, moves it anywhere from 1.1x to 2.9x. Staff in Slack polls say they are working 12 times faster. And most of the evidence that the speedup comes from AI was redacted.

Two of Ten Were Trained for the Test

14 minute read

· Benchmarks & measurement

Memorilla, from a Stanford-led group, packs a whole document collection into 16 soft tokens for a frozen Qwen3-8B and posts the best average score across eight benchmarks. Its biggest win is 44.60% against 30.90% on PersonaMem-v2. Read the appendix, though, and of the ten methods in the main table only two were trained on the benchmarks’ training splits: Memorilla and xRAG, the runner-up on that benchmark. The other eight, Mem0 and MemoRAG included, ran as shipped.

The Top Three Tax Agents Are Claude. So Is the Grader.

15 minute read

· Benchmarks & measurement

Surge AI’s new DAYJOB: Tax benchmark puts three Claude models on top, led by Sonnet 5.5 at 10.8 percent. The harness that produced those numbers grades every attempt with Claude Opus 4.8. Surge’s own launch post for the benchmark family named model-family bias as a weakness of LLM judges. At 250 attempts per model, the top three are also too close to rank.

One Crawlable Page in Four Has AI Text. At the Center of the Web It Is One in Nine.

7 minute read

· Benchmarks & measurement

A new study of more than 5 million crawlable pages estimates that about 24% carried AI-generated or AI-edited text in August 2026. The rate is 11.6% in the web’s most central tier and 27.5% at its periphery, which holds most of the pages and nearly four-fifths of the growth. It is about 6% on .gov and .edu, and twice as high in English as in other languages.

Harder to Read Than People, and Easier Than Last Time on 19 of 21 Counts

7 minute read

· Benchmarks & measurement

A new paper finds that frontier models write with more of the features that slow human readers than people do. Its starting point is a sense that AI text has recently become harder to read, but its measurements compare models with people at one moment. The one before-and-after pair it reports, Claude Opus 5 to Opus 5.5, moved toward easier text on 19 of 21 comparisons.

Mistral Large 4’s Cyber Lead Comes From the Test Its Rivals Declined

16 minute read

· Benchmarks & measurement

Mistral launched a preview of Large 4 on 6 October with a cyber headline: 82% on a test where Claude Opus 5.5 and GPT-6 Astra score near zero because they refuse. Artificial Analysis’s own data confirms the refusals, and also shows that on the two tests in the same index where every model takes part, the preview trails both. The lead over the closed models is mostly a refusal effect, and the checkpoint that earned it cannot be downloaded yet.

Claude Leads 26% of Anthropic’s Research. The People Who Own That Work Agreed With Each Other 35% of the Time.

19 minute read

· Benchmarks & measurement

Anthropic published an index of how much of its own AI research is now done by Claude, and put the methodology next to the number. The methodology is the interesting part. Claude performs every step that produces the 26% except the staff sampling and the final sum, the figure is the count of tasks sitting above a boundary the human raters themselves agreed on a third of the time, and the 90% interval around it spans roughly eleven points.

Sixteen Tokens Held the Whole Strategy. Thirty-Eight of One Hundred and Two Runs Overfit Anyway.

18 minute read

· Benchmarks & measurement

Amazon’s research arm says machine-learning research agents don’t overfit the validation set they hammer, and that compressibility explains it. The experiment that supports the headline also contains a control where 38 of 102 runs scored more than 10 percent better on validation than on the holdout. The difference between the two conditions is not the agents. It is whether the harness let them near the data.

Gemini’s Voice-Agent Lead Is 0.7 Points, Over a Rival Scored on One Trial

21 minute read

· Benchmarks & measurement

Google says Gemini 3.8 Live Extended Thinking leads the voice-agent tables at 68.6% on Ο„-Voice and 35.1% on a banking benchmark. The lead over OpenAI on Ο„-Voice is 0.7 points, and the independent evaluator scored OpenAI’s model on one trial. The cheaper model that shares the name, and goes into Search, completes 30.1% of the same tasks. Neither score says how often a caller gets the right outcome twice.

Two of the Seven Judges Were Claude Code

18 minute read

· Benchmarks & measurement

A tracker published this week asked seven frontier AI agents to recommend products across 161 categories, and found that models favour their own maker’s tools. They do. But the seven agents are seven model-CLI configurations, two of them Claude Code, and the study’s own methodology page says the harness is part of what was measured. The write-up that carried the finding dropped that, and turned 28 of 61 eligible categories into 28 of 161.

page 1 of 2 · 20 essays on Benchmarks & measurement