Four Kenwood High School students sued Omnilert, Baltimore County Public Schools and county police this week over an October 2025 night when an AI gun detector flagged a bag of Doritos and armed officers handcuffed them. The complaint calls the systemβs 99 percent false-positive rate the failure. Marylandβs inspector general measured something narrower, and the alarm-fatigue literature puts that number in familiar company. What failed that night was who could act on an alert after it had been canceled.
Dario Caldara and JesΓΊs FernΓ‘ndez-Villaverde ran 25 real referee reports through an AI detector, then had two chatbots proofread, edit, rewrite and replace them. The detector called every original human, and every report written from scratch recommended major revision of an already published paper. The second result is the stronger one. Twenty-five clean reports can only rule out a false-positive rate above about 11 percent, and that bound matters a great deal once a journal starts acting on flags.
A preregistered field experiment by Saharsh Agarwal and Ananya Sen hid Googleβs AI Overviews from randomly chosen desktop users and found outbound clicks on affected searches fell from 0.62 to 0.37 per search, a 39.8 percent cut. That is the number in the headlines. The number a publisherβs analytics will show depends on how many searches get an Overview, and the published estimates of that share run from 18 to 67 percent.
A new synthesis of nine multi-agent studies finds that once the single model gets the same number of samples or the same budget, the multi-agent systems average 4.2 points worse and come out ahead in about a quarter of comparisons. In one study, five agents debating beat a single answer by 4.5 points, and five answers with no conversation at all beat it by 7.6. Researchers studying human groups hit the same baseline problem decades ago.
METR spent eleven weeks inside Anthropic and concluded that capability progress there has roughly doubled since October 2025. The headline figure is a 1.9x acceleration on Anthropicβs version of the Epoch Capabilities Index. The reportβs own appendix shows that one modeling choice, which the data cannot settle, moves it anywhere from 1.1x to 2.9x. Staff in Slack polls say they are working 12 times faster. And most of the evidence that the speedup comes from AI was redacted.
Memorilla, from a Stanford-led group, packs a whole document collection into 16 soft tokens for a frozen Qwen3-8B and posts the best average score across eight benchmarks. Its biggest win is 44.60% against 30.90% on PersonaMem-v2. Read the appendix, though, and of the ten methods in the main table only two were trained on the benchmarksβ training splits: Memorilla and xRAG, the runner-up on that benchmark. The other eight, Mem0 and MemoRAG included, ran as shipped.
Googleβs Gemini 4 Carbon announcement reports 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench. Only the last appears on its benchmarkβs public leaderboard, where Carbon leads by three points over a 120-task set. The safety report the post links to is the one for Gemini 4 Argon, and it never mentions Carbon.
Surge AIβs new DAYJOB: Tax benchmark puts three Claude models on top, led by Sonnet 5.5 at 10.8 percent. The harness that produced those numbers grades every attempt with Claude Opus 4.8. Surgeβs own launch post for the benchmark family named model-family bias as a weakness of LLM judges. At 250 attempts per model, the top three are also too close to rank.
A new study of more than 5 million crawlable pages estimates that about 24% carried AI-generated or AI-edited text in August 2026. The rate is 11.6% in the webβs most central tier and 27.5% at its periphery, which holds most of the pages and nearly four-fifths of the growth. It is about 6% on .gov and .edu, and twice as high in English as in other languages.
A new paper finds that frontier models write with more of the features that slow human readers than people do. Its starting point is a sense that AI text has recently become harder to read, but its measurements compare models with people at one moment. The one before-and-after pair it reports, Claude Opus 5 to Opus 5.5, moved toward easier text on 19 of 21 comparisons.
Mistral launched a preview of Large 4 on 6 October with a cyber headline: 82% on a test where Claude Opus 5.5 and GPT-6 Astra score near zero because they refuse. Artificial Analysisβs own data confirms the refusals, and also shows that on the two tests in the same index where every model takes part, the preview trails both. The lead over the closed models is mostly a refusal effect, and the checkpoint that earned it cannot be downloaded yet.
Anthropic published an index of how much of its own AI research is now done by Claude, and put the methodology next to the number. The methodology is the interesting part. Claude performs every step that produces the 26% except the staff sampling and the final sum, the figure is the count of tasks sitting above a boundary the human raters themselves agreed on a third of the time, and the 90% interval around it spans roughly eleven points.
Amazonβs research arm says machine-learning research agents donβt overfit the validation set they hammer, and that compressibility explains it. The experiment that supports the headline also contains a control where 38 of 102 runs scored more than 10 percent better on validation than on the holdout. The difference between the two conditions is not the agents. It is whether the harness let them near the data.
Google says Gemini 3.8 Live Extended Thinking leads the voice-agent tables at 68.6% on Ο-Voice and 35.1% on a banking benchmark. The lead over OpenAI on Ο-Voice is 0.7 points, and the independent evaluator scored OpenAIβs model on one trial. The cheaper model that shares the name, and goes into Search, completes 30.1% of the same tasks. Neither score says how often a caller gets the right outcome twice.
A tracker published this week asked seven frontier AI agents to recommend products across 161 categories, and found that models favour their own makerβs tools. They do. But the seven agents are seven model-CLI configurations, two of them Claude Code, and the studyβs own methodology page says the harness is part of what was measured. The write-up that carried the finding dropped that, and turned 28 of 61 eligible categories into 28 of 161.
page 1 of 2 · 20 essays on Benchmarks & measurement