Models, inference & pricing, page 2

A Lot More B Than A

15 minute read

· Models, inference & pricing

A local inference engine for DeepSeek V4 Flash hit Hacker News again today, credited to the creator of Redis. The link actually goes to a fan-built explainer site, and the live argument in the comments over whether its aggressive quantization holds up cites a benchmark as proof. The benchmark never mentions the project, and its own author says the two runs in it can’t answer that question.

437x Against a 2023 Baseline: What the DeepSeek Cache Number Does and Doesn’t Show

11 minute read

· Models, inference & pricing

A blog post making the rounds says Western labs quietly copied DeepSeek’s cache trick, and offers two price cuts as proof. The 437x figure is real arithmetic from DeepSeek’s own paper. The proof is a different matter: both price cuts also fit a simpler change on the price sheets, and neither lab’s launch post mentions KV caches or DeepSeek.

Medium Is Not a Unit

16 minute read

· Models, inference & pricing

Anthropic’s new prompting guide for Claude Opus 5.5 says the quiet part plainly: effort level names don’t mean the same amount of thinking from one model to the next. The guide backs its advice with nine findings from Anthropic’s own testing. Eight give a direction, one gives a size, and none gives a sample. A sibling page on the same site shows what the missing numbers look like, and they complicate the advice.

Three Million Dollars for the Last Stage

12 minute read

· Models, inference & pricing

A headline said Xiaomi’s trillion-parameter MiMo-V2.6-Pro was trained for $3M. Xiaomi’s own report says something narrower: $2.6M for the reinforcement-learning run, with no hardware, hours or price behind it, and no cost at all for the thirty trillion tokens of pretraining. That narrower number is still the interesting one. The final stage of training now costs about nine times what DeepSeek spent on R1’s entire reasoning training.

Z.ai Named the Kernel Flag. It Did Not Name the Chip.

21 minute read

· Models, inference & pricing

Z.ai says it runs all production inference for one of its models on more than 100,000 Chinese-made AI accelerators, at a per-token cost comparable to NVIDIA. The post names the quantization scheme, the Triton precision flag, and the exact library version whose C++ bindings held the Python GIL. It never names the accelerator. The most obscure claim in it is the only one I could fully verify.

The Biggest Language Model Is 2.8 Trillion Parameters. 83 Percent of Downloads Go to Models Under One Billion.

14 minute read

· Models, inference & pricing

Moonshot’s Kimi K3 is the largest disclosed language model on the market, at 2.8 trillion parameters. Nobody outside OpenAI, Anthropic, or Google knows how big GPT-6 Astra, Claude, or Gemini actually are, because none of the three labs discloses a parameter count. Meanwhile Hugging Face’s own numbers say 83 percent of all downloads on its Hub go to models under a billion parameters, and only 1 percent go above a hundred billion. Three different questions, three different answers, and none of them is β€˜how big is AI.’

Eight Billion Parameters, Five Hundred and Ten Gigabytes

15 minute read

· Models, inference & pricing

DeepSeek released V4.1-Flash this morning. The card’s first sentence says 552B parameters. Three paragraphs later it discloses another 196B of memory. The checkpoint on disk holds 763.2B, and the model touches 8B of them to read a token. Every one of those numbers is correct, and the industry still prints one column.

The Router Collapsed, Not the Experts

20 minute read

· Models, inference & pricing

Two preprints went up a day apart this month and were read as proof that the load-balancing loss fills mixture-of-experts models with redundant experts you can delete for free. One of them says the opposite in plain type. The homogenization is real, it was measured in a controlled run back in May, and it lands on the router rather than the experts. In that experiment the loss whose entire job is balancing load balanced it six times worse than not having the loss at all.

Synthetic Data Doesn’t Poison Models. Throwing Away the Real Data Does.

20 minute read

· Models, inference & pricing

Model collapse is real, reproducible, and provable in the limit β€” but every canonical demonstration of it replaces the human data with machine output. Change that one assumption and the theory changes with it: accumulate real and synthetic data and the test error is provably bounded; filter synthetic data through a verifier and it beats human data outright. What degrades in every regime, including the successful ones, is diversity.

page 2 of 2 · 27 essays on Models, inference & pricing