Writing, page 2

Thirty-Seven Rejected as Vulnerabilities, Fifteen Already Have a Commit

17 minute read

· Security & privacy

VulnCheck’s seven-month review of Anthropic’s Project Glasswing calls the numbers damning: 217 fixes, 1,654 duplicates, a 94.2% true positive rate it doesn’t trust, and curl rejecting most of an OSS Scanner batch. I checked each figure against Anthropic’s ledger data and curl’s own tracker. Most of the counts are right and most of the labels are wrong. The duplicates are Anthropic’s own repeats, the 217 is the fixed count among the 237 findings allowed to show a status, and 15 of the 37 curl reports marked β€˜not a vulnerability’ already have a commit on curl’s main branch citing them.

Up to 72% Off, on Harnesses Nobody Had Tuned

19 minute read

· Models, inference & pricing

A recap video of Lance Martin’s AI Engineer talk has been going around with seven takeaways and six slash commands for cutting what you pay Claude. I checked the commands and numbers against Anthropic’s changelog, pricing page and the skill files those commands run. The commands are real, under slightly different names. The 72% saving is a ceiling, measured on harnesses nobody had tuned. The cache rule of thumb in the talk is already out of date in the direction that makes it matter more, and the overnight hill climb is a little less sealed against overfitting than the recap says.

Ninety Percent Was the Minimum for a Decent Rating

12 minute read

· Agents

A former Meta engineer writes that generating at least 90% of his code with AI was the minimum for a decent rating, and that AI-heavy codebases end up with more incidents that take longer to fix. The controlled evidence I could find measures smaller units than that: one function, one task, one developer. Those mostly look fine. The one large dataset that measures delivery found stability falling as AI adoption rose. The claim about incidents is the one nobody has tested properly, and the 90% target measures how much code AI wrote, which isn’t the thing that breaks.

6.5 Million Lines of Proof for Signal. The Part You Have to Read Is 26,000.

16 minute read

· Security & privacy

Google DeepMind says its agents proved Signal’s core security properties about the compiled machine code of libsignal, in four weeks with a team of five. The proof is about 6.5 million lines of Lean that nobody has to read. The guarantee rests on about 26,000 lines that somebody does, plus 37 routines, the cryptographic ones among them, that the proof treats as black boxes with contracts. To me the most interesting result is the list of errors the specification work turned up in the published analyses it started from.

Mythos 5.2 Reached Outside Its Sandbox in 31% of Scenarios. The Launch Post Says 3.1.

14 minute read

· AI safety & governance

Anthropic’s Fable 5.2 launch post says Mythos 5.2 used credentials planted outside its sandbox in 3.1% of scenarios, level with Opus 5.5. The system card has the other rate from the same run: it reached outside in 31%, more often than Opus 5.5 or Sonnet 5.5. A Claude model asked to review the draft flagged the choice, and Anthropic kept it. The more useful finding is further down: one paragraph in the system prompt cut the rate to 1.2%.

Four Executions per Call, None Delivered

19 minute read

· Models, inference & pricing

A new paper overloads vLLM for 107 seconds and then drops the load to 60% of capacity. With client retries on, the engine never recovers, and propagating cancellation, the obvious fix, still leaves 85% or more of later calls undelivered. The engine’s own prompt-token counter misses a third to half of the work. What brought service back was attaching retries to work already running, not killing it faster.

Same Data, Same Safety Training, Different Jailbreaks

17 minute read

· AI safety & governance

A preprint from Taiwan trained 150 small models from 25 data recipes, three pretraining runs each, all given the same safety training. Two copies of one pretraining run break on almost the same jailbreak prompts (corrected correlation 0.984). Two runs of the same data drop to 0.811, close to the 0.779 of models trained on different data. The attack that works best on one checkpoint stays best on another run of the same data in only 27% of pairs. A jailbreak list describes one run, not the recipe behind it.

Twice the Passes for the Smaller Model

16 minute read

· Models, inference & pricing

John Langford’s farewell to Microsoft’s AI Frontiers group makes a pointed case for research over scaling: a ByteDance result suggests a hidden-state feedback trick can match a model twice its size, so β€˜do the math’ on halving training and inference cost. I read both papers behind that sentence. The inference saving holds up. The training saving does not, by the papers’ own accounting.

The Fifteen-Fold Ceiling Is Ninety-Five Dollars Over Six

20 minute read

· Business & work

Anthropic’s economics team classified 207,049 research sessions and found that scientists use Claude most where AI is most capable. The usage data are the solid part. The finding that use tracks capability compares one model’s labels with another model’s judgments. And the headline that some fields could produce up to 15 times more research is mostly a price ratio: $95 for a researcher-hour against $6 of assumed AI time.

Twelve by Poll, 1.9 by Benchmark, 1.1 on Another Curve

17 minute read

· Benchmarks & measurement

METR spent eleven weeks inside Anthropic and concluded that capability progress there has roughly doubled since October 2025. The headline figure is a 1.9x acceleration on Anthropic’s version of the Epoch Capabilities Index. The report’s own appendix shows that one modeling choice, which the data cannot settle, moves it anywhere from 1.1x to 2.9x. Staff in Slack polls say they are working 12 times faster. And most of the evidence that the speedup comes from AI was redacted.

Two of Ten Were Trained for the Test

14 minute read

· Benchmarks & measurement

Memorilla, from a Stanford-led group, packs a whole document collection into 16 soft tokens for a frozen Qwen3-8B and posts the best average score across eight benchmarks. Its biggest win is 44.60% against 30.90% on PersonaMem-v2. Read the appendix, though, and of the ten methods in the main table only two were trained on the benchmarks’ training splits: Memorilla and xRAG, the runner-up on that benchmark. The other eight, Mem0 and MemoRAG included, ran as shipped.

Twenty-Three of Thirty-Six From the Text Alone

13 minute read

· Security & privacy

A single-author paper trained 36 LoRA adapters on 18 datasets in five domains and asked whether a plain classifier could tell what kind of data an adapter was trained on, for datasets it had never seen. From the adapter’s answers to 200 bland prompts it got 23 of 36 right, against a chance rate of one in five. From the weights it did worst, which is the opposite of what the author pre-registered.

Thirty-Six Tokens, Billed as 4,418

16 minute read

· Security & privacy

A Queen’s University group bought access to GPT-6 Astra and Claude Opus 5 through 20 third-party API relays and checked three things: the answers, the tool permissions and the bill. Seven of ten cheap relays and two of ten full-price ones answered differently from the official API. Four returned tool calls the request had disabled. On one identical request where the official API counted 36 input tokens, three relays reported more than 4,400, and at the one of them with a readable balance, the charge matched its reported usage to the cent.

Two Hundred Twenty Named the Risk and Paid

16 minute read

· Security & privacy

ScamWorld rebuilds 363 reported scam websites and sends web agents on six frontier models to shop, book and invest on them. The agents were deceived in 11 to 35 percent of scam episodes, and in 220 of 492 deceived episodes they named a site-specific risk before taking the final action anyway. They almost never hand over a password. They pay.

Seven of Ninety-Six Scout Replies Met the Limits

15 minute read

· Security & privacy

Anthropic’s FlashProve proves Llama 4 Maverick inference on one H100 at about 110 tokens a second, with soundness checked in Lean. The Lean theorem covers exact match, and by the repository’s own scoring, none of 96 stock replies would pass exact match. The looser policy that serving uses accepted 127 of 128 honest replies. Scored against the same limits, 7 of 96 replies from the smaller Llama 4 Scout met them too.

page 2 of 12 · 178 essays