Models, inference & pricing

Essays on open weights, inference, quantization, hardware and what a model costs per task rather than per token.

Twice Flash’s Price Per Token, a Quarter of Its Cost Per Task

15 minute read

· Models, inference & pricing

Gemini 4 Hydrogen lists at $1.50 and $7.50 per million tokens, twice what Gemini 3.8 Flash charges today and exactly what Flash charges from 1 January. On Artificial Analysis’s index runs it still costs about a quarter as much per task, because it reads about half the input. The 2.5x speed claim holds up on outside data. The β€˜available to everyone’ line holds on the API and in Search, and less so in the free Gemini app.

Up to 72% Off, on Harnesses Nobody Had Tuned

19 minute read

· Models, inference & pricing

A recap video of Lance Martin’s AI Engineer talk has been going around with seven takeaways and six slash commands for cutting what you pay Claude. I checked the commands and numbers against Anthropic’s changelog, pricing page and the skill files those commands run. The commands are real, under slightly different names. The 72% saving is a ceiling, measured on harnesses nobody had tuned. The cache rule of thumb in the talk is already out of date in the direction that makes it matter more, and the overnight hill climb is a little less sealed against overfitting than the recap says.

Four Executions per Call, None Delivered

19 minute read

· Models, inference & pricing

A new paper overloads vLLM for 107 seconds and then drops the load to 60% of capacity. With client retries on, the engine never recovers, and propagating cancellation, the obvious fix, still leaves 85% or more of later calls undelivered. The engine’s own prompt-token counter misses a third to half of the work. What brought service back was attaching retries to work already running, not killing it faster.

Twice the Passes for the Smaller Model

16 minute read

· Models, inference & pricing

John Langford’s farewell to Microsoft’s AI Frontiers group makes a pointed case for research over scaling: a ByteDance result suggests a hidden-state feedback trick can match a model twice its size, so β€˜do the math’ on halving training and inference cost. I read both papers behind that sentence. The inference saving holds up. The training saving does not, by the papers’ own accounting.

Ninety-Three Percent of the Answers Never Ended

13 minute read

· Models, inference & pricing

An MIT group had GPT-6 Astra edit 0.016 percent of Qwen3.5-0.8B’s weights, and MATH-500 went from 7.9 to 48.1 percent. Before the edit, 93 percent of the model’s answers ran into the 32,768-token limit, in a setup its own model card warns about. The edit taught the model to stop, which is a real repair but a narrower one than better math, and it came from one of seven runs.

Two Percent of the Weights, None of Them Trained

15 minute read

· Models, inference & pricing

nesformer runs a Nintendo’s 6502 CPU inside an ordinary Qwen3.5 checkpoint. The headline says about 1GB of weights, but only 9.3 million of the 466.6 million parameters are nonzero, the picture is drawn outside the model, and it runs 150 to 290 times slower than the console. The part worth keeping is a different one: the weights are the machine, and the game lives in the model’s state, which is somewhere nobody inspecting a model file would look.

A 75% Price Cut That Leaves the Chat Bill Unchanged

7 minute read

· Models, inference & pricing

Frontier API prices have fallen by up to 80% since June, Max Woolf writes. The headline cuts land on different parts of the bill. Fable 5.1’s cache reads are 75% cheaper while its input and output prices did not move, so a chat workload pays the same as before. On an agent workload I worked through, Opus 5.5 is about 28% cheaper than Opus 5, not 40 to 60.

Twelve Steps, Coarse to Fine

7 minute read

· Models, inference & pricing

Hazy Research’s Next-Concept Modeling generates text in a fixed hierarchy of short-to-long code sequences instead of 128 denoising passes over the whole thing. The paper reports 10 to 31 times faster generation than diffusion baselines. The speedup is real in the tables, and the quality claims need reading one row at a time.

The EmbeddingGemma 2 Model Card Has One Baseline. The Launch Charts Have Twenty-One More.

16 minute read

· Models, inference & pricing

Google’s EmbeddingGemma 2 is described as best in class for its size. As of 14 October 2026, its model card compares it with exactly one model, its predecessor, and reports a flat multilingual score. The comparisons that support the headline sit in three chart images in the launch post, and the license line says Apache 2.0 and also asks for adherence to a Gemma policy.

Open Is a Promise Until the File Ships

12 minute read

· Models, inference & pricing

The Financial Times says Nvidia is in early talks to buy Reflection AI, five days after Reflection announced its first open-weight model. As of 10 October the weights, the license file and the technical report are all still promises.

What a Tenth of a Bit Buys

8 minute read

· Models, inference & pricing

Samsung’s LittleBit paper compresses weights to 0.1 bits each. That number is counted one layer at a time. The memory saving for a whole model is about 21 times on Llama2-7B, not 160, and the quality cost at that budget is steep.

What 4.4 Percent Active Doesn’t Measure

11 minute read

· Models, inference & pricing

Aleph Alpha released Kolibri-1 today, a 78B-parameter model that activates 3.46B per token, billed as Germany’s sovereign open model. Its own report says it leads the 3B-active class by about a point, and its headline efficiency chart prices decode throughput but not the 78 GB that has to sit in memory.

page 1 of 2 · 27 essays on Models, inference & pricing