Twice the Passes for the Smaller Model

On 6 November John Langford posted “The End of AI Frontiers” on his blog, hunch.net, and by the next day it was on the Hacker News front page. AI Frontiers was a group of about 50 people that he and others set up inside Microsoft Research in 2023. In early September, he writes, it was “reorged into Microsoft AI,” whose leadership came over from Inflection. He describes that organization’s philosophy this way: “The core philosophy there is scaling, and aggressively not research.” He has since left.

Most of the post is about how research organizations work, and I have nothing to add to his account of his own employer. The paragraph I kept rereading is the one where he puts a number on the case for research:

“For example, the Bytedance experiments suggest a variant of the Full Bandwidth Transformer might be comparable to a model of twice the size, implying significant savings in training and inference for the same capability. Do the math—how much is it worth to halve training and inference cost given that the industry has investments approaching a trillion dollars?”

That sentence points to two papers: the Full-Bandwidth Transformer, which Langford co-authored, and ByteDance Seed’s Recycler. Both are public, so I read them to see what doing the math gives.

One idea, two papers

When a standard transformer generates text, each step ends with a top-layer hidden state, a long vector holding everything the model computed at that position. That vector is turned into a probability distribution, one token is sampled, and the next step starts from that token’s embedding. The vector itself is thrown away. Both papers keep it.

The Full-Bandwidth Transformer (FBT), posted to arXiv on 9 August and revised on 1 October, fuses the previous top-layer state with the next token’s embedding through a gated linear unit and feeds the result back in. The authors say the fusion “adds two matrix multiplications per generated token,” which they put at under 1% of a forward pass. Recycler, dated 25 October, does nearly the same thing with a simpler adapter: an RMS norm and a linear projection added to the token embedding.

Langford calls Recycler “a variant” of FBT. ByteDance files it under concurrent work: “Recycler is closer to the latter in injection site, but uses a simpler additive adapter.” The difference matters little for the economics. Both designs face the same problem and solve it the same way.

That problem is training. If step t needs the hidden state from step t−1, you can’t process a whole sequence in parallel, and parallel training is the reason transformers are cheap to train. So both papers approximate the recurrence with several full passes over the sequence. Each pass takes the previous pass’s hidden states, shifts them one position, and runs again. That works, and it costs a forward and backward pass per extra pass. The cost lands on the training side.

Where hidden-state feedback costs and where it saves Comparison of the two designs. In standard decoding the top-layer hidden state is reduced to a sampled token and discarded. In feedback decoding, used by both the Full-Bandwidth Transformer and ByteDance's Recycler, the hidden state is also passed to the next step through a small adapter. FBT puts the decoding overhead at two matrix multiplications per token, under 1% of a forward pass, and Recycler describes its decoding cost as approximately the same as a standard transformer. Training needs multiple passes over each sequence: Recycler says its training cost is 2 times a standard model's, plus 3.3% for a partial third pass, and FBT reports 1.28 to 1.5 times token-equivalent compute for its 400 billion token runs. Prefill with an extra fused pass doubles prefill compute in both papers. ONE DECODING STEP · STANDARD VS HIDDEN-STATE FEEDBACK · FBT AND RECYCLER Standard decoding Feedback decoding Hidden state Sample token Embed Hidden state Sample token Embed + h The vector is reduced to one token, then discarded. Previous h re-enters through a small adapter Decoding FBT: two matrix multiplications per token, under 1% of a pass. Recycler: about the same cost as a standard transformer. Prefill Same as standard with one pass; doubled with an extra fused pass (both papers). Training Recycler: 2× a standard model, +3.3% for a partial third pass. FBT: 1.28× to 1.5× token-equivalent compute on its 400B-token runs. Sources: Wang et al., arXiv:2608.08888 (v2, 1 Oct 2026); ByteDance Seed, Recycler (25 Oct 2026).
Both designs are nearly free when the model generates text. Training is where they cost more.

What “twice the size” measured

Recycler’s abstract says the method ends up “outperforming models with 2–3× more parameters.” The comparison behind that is in the pretraining section. ByteDance trains a model called modelβ, a baseline 2.4 times its size, and a same-size Recycler. At the end of the displayed training window, Recycler’s average across 14 benchmarks is 64.76. The 2.4x baseline scores 66.43.

So the bigger model still wins, by under two points. For a model less than half the size, that’s impressive. It still isn’t a model “comparable to a model of twice the size” in the sense of a controlled match. The paper says so itself. The same-size baseline’s run stops short, “ending near 50% of the shared display range,” and “we therefore show it without extrapolation and do not treat the three endpoints as an equal-training-budget comparison.” The authors call the central comparison “computational”: Recycler “nearly matches the substantially larger model while preserving the decoding cost of the smaller model.”

The cleaner like-for-like test is on a second model, modelα, where baseline and Recycler share a backbone, data and recipe. There the 14-benchmark average goes from 42.97 to 46.21. The paper does not give the parameter counts of modelα or modelβ. The only size it states is for a small extrapolation test (151 million activated parameters out of 1 billion total).

Recycler's pretraining comparisons Equal-weight average across 14 downstream benchmarks after pretraining, from ByteDance Seed's Recycler paper dated 25 October 2026. modelα, a controlled comparison with shared backbone, data and recipe: baseline 42.97, Recycler 46.21. modelβ, a scaling comparison that the paper says is not an equal-training-budget comparison: Recycler 64.76 against a baseline with 2.4 times the model size at 66.43. The same-size modelβ baseline's run ends near 50% of the training range, so it has no endpoint score. 14-BENCHMARK AVERAGE AFTER PRETRAINING · RECYCLER, BYTEDANCE SEED, 25 OCT 2026 modelα · controlled: same backbone, data, recipe modelβ · scaling comparison, not equal budget Baseline Recycler Baseline, 2.4× size Recycler, 1× size 42.97 — Recycler paper, modelα baseline 46.21 — Recycler paper, modelα Recycler 66.43 — Recycler paper, modelβ Baseline ×2.4 64.76 — Recycler paper, modelβ Recycler 42.97 46.21 66.43 64.76 The same-size modelβ baseline stops near 50% of the training range and has no endpoint score. The paper does not state parameter counts for modelα or modelβ.
The 2.4x-larger model still comes out ahead by under two points, and the paper declines to treat that comparison as equal-budget.

The training bill

This is where “halve training cost” stops matching the papers. Recycler’s pretraining section says it directly: “The training cost of Recycler is 2× that of the standard model.” Larger runs also needed a third pass on some steps to keep the recurrence stable, at “an additional 3.3% in training cost.” The model is smaller, but each token it trains on costs about twice as much.

ByteDance’s own cost chart makes the trade explicit. It compares turning Recycler on at three stages of training, each against a matched baseline, and plots score gain against estimated cumulative training compute. Turning it on for all of pretraining adds 100.50% to the compute bill for +1.87 points. Turning it on only during continued pretraining adds 13.54% for +0.60. Turning it on only at fine-tuning adds 0.19% for +1.14. The authors call full pretraining adoption “prohibitively high” in overhead and use the continued-pretraining route for their headline models.

Recycler's cost and gain by the stage where it is introduced From Figure 8 of ByteDance Seed's Recycler paper, 25 October 2026, each route compared with a stage-matched baseline. Introduced at pretraining: estimated cumulative training compute up 100.50%, benchmark score up 1.87 points. Introduced at continued pretraining: compute up 13.54%, score up 0.60. Introduced at supervised fine-tuning: compute up 0.19%, score up 1.14. The fine-tuning comparison uses a different benchmark suite from the other two, so scores are compared only within each pair. RECYCLER VS MATCHED BASELINE · BY STAGE INTRODUCED · FIG. 8, 25 OCT 2026 Extra training compute Score gain, points From pretraining From continued pretraining From fine-tuning only +100.50% compute — Recycler paper, Fig. 8 +13.54% compute — Recycler paper, Fig. 8 +0.19% compute — Recycler paper, Fig. 8 +1.87 points — Recycler paper, Fig. 8 +0.60 points — Recycler paper, Fig. 8 +1.14 points — Recycler paper, Fig. 8 +100.50% +13.54% +0.19% +1.87 +0.60 +1.14 Compute is estimated cumulative training FLOPs relative to the pretraining baseline. The fine-tuning row uses a different benchmark suite; compare scores only within a row.
Using the method through all of pretraining roughly doubles the training bill, which is why ByteDance's headline models only switch it on later.

None of this makes Recycler a bad idea. The fine-tuning route is close to free and still helps, and after reinforcement learning the best configuration scores 64.38 against the baseline’s 60.06. But ByteDance’s claim is a cheaper model to run, and its own pretraining section says that model costs more to train.

FBT’s own numbers

The Full-Bandwidth Transformer paper counts training compute more carefully than most architecture papers do. It defines “token-equivalent compute” as training tokens times the average number of forward passes per batch, and it lists that figure for every run. The trick behind its efficiency is a schedule: in every run past 10B tokens, the first 75% of training uses ordinary single-pass batches, and the extra feedback passes are added only near the end.

FBT training tokens and token-equivalent compute From Table 1 of the Full-Bandwidth Transformer paper, arXiv 2608.08888, revised 1 October 2026, for 1-billion-parameter models. 10 billion tokens, all three-pass: 30 billion token-equivalent compute. 100 billion tokens, 75% one-pass and 25% three-pass: 150 billion. 200 billion tokens, 75% one-pass, 22% two-pass, 3% three-pass: 256 billion. 400 billion tokens on the same schedule: 512 billion, the 1.28 times run. 400 billion tokens with 75% one-pass and 25% three-pass: 600 billion, the 1.5 times run. FBT, 1B PARAMETERS · TOKENS VS TOKEN-EQUIVALENT COMPUTE · TABLE 1, ARXIV:2608.08888 10B tokens 100B tokens 200B tokens 400B tokens (1.28×) 400B tokens (1.5×) 10B tokens — FBT Table 1 100B tokens — FBT Table 1 200B tokens — FBT Table 1 400B tokens — FBT Table 1 400B tokens — FBT Table 1 30B token-equivalent — FBT Table 1 150B token-equivalent — FBT Table 1 256B token-equivalent — FBT Table 1 512B token-equivalent — FBT Table 1 600B token-equivalent — FBT Table 1 10B 100B 200B 400B 400B 30B 150B 256B 512B 600B Light bar: training tokens. Dark bar: tokens × average forward passes per batch.
Because the extra passes come late in training, FBT's 400B-token runs cost 1.28x to 1.5x a standard run rather than 2x or 3x.

The paper summarizes its benefit in two ways. The abstract says full-bandwidth models “match or approach standard transformers trained with roughly 1.5× more tokens.” The introduction is stronger: with extra prefill passes, the model “matches standard transformer baselines trained on twice the tokens” on validation loss and multiple-choice accuracy, and on some generation tasks it “approaches baselines trained with up to 5× the tokens.”

My arithmetic uses only the paper’s numbers and assumes training cost scales with token-equivalent compute. On the abstract’s reading, a run costing 1.28x to 1.5x roughly matches a baseline costing 1.5x. That ranges from a saving of about 15% to break-even. Take the introduction’s sentence at face value and a run like the 200B one, at 256B token-equivalent, does what a 400B standard run does, for about 36% less compute. Neither reading halves training cost. The data story is better. The paper’s footnote notes that its baselines saw the same data “since there is no extra corpus available,” and a method that does what more tokens would do matters when you run out of tokens. Still, that’s a different claim from cheaper.

The FBT models are also 1 billion parameters. Langford’s trillion-dollar framing applies to frontier models, and neither paper tests one. ByteDance’s results are the bigger-scale evidence, and ByteDance doesn’t say how big its models are.

The part that survives

Inference is where Langford’s point holds. A model that nearly matches one 2.4 times its size, with “the same token-by-token decoding cost and KV-cache footprint as its same-size baseline,” saves money on every token served for as long as the model is deployed. Training is paid once; serving is paid every day. If one model generation of Recycler-style feedback roughly halved the serving cost of a given capability level, that alone would justify a research group of 50 people many times over.

The trade-off has a familiar shape. The Chinchilla paper showed that for a fixed training budget, model size and training tokens should be scaled together, and its authors noted that a smaller, longer-trained model “uses substantially less compute for fine-tuning and inference.” Spending more on training to get a smaller model that is cheaper to serve is an old move. Hidden-state feedback is a new lever for it: extra passes in place of extra tokens. Recent theoretical work derives the compute-optimal rule from a model of how concepts are learned from text, so the rule describes a way of training rather than a fixed property of language models.

The recurrence itself has a long history too. A recent survey of recurrent models in the transformer era describes a “strong recent revival” of recurrent computation, and both papers fit that pattern: a parallel model with a recurrent channel added at the one point where generation is already sequential.

What I’d take from it

Langford’s argument against a scaling-only organization doesn’t need the word “halve.” The evidence he cites shows a modest architectural change doing work that would otherwise take more parameters or more data, at almost no extra decoding cost, in two labs working separately: in ByteDance’s results, a model approaching one 2.4 times its size; in FBT’s, a 1B model matching baselines trained on more tokens. A shop that only scales is unlikely to find results like that, because nobody there is paid to look for them.

But the sentence as written claims two savings and the papers support one. Training gets more expensive per token, by 2x for Recycler if used from the start and 1.28x to 1.5x for FBT with its late schedule, and the training saving shrinks to somewhere between nothing and about a third depending on which of FBT’s summaries you use. The bigger-model comparison is not equal-budget, by ByteDance’s own description. And the clean results are at 1 billion parameters or at undisclosed sizes.

Two things would settle it. One is a run at a disclosed, large scale where baseline and feedback model get the same total training FLOPs and the benchmark gap is reported. The other is a serving-cost comparison at equal quality, in dollars or GPU-hours per million tokens. If the second comes out near half, Langford’s trillion-dollar arithmetic works for inference alone.

References

  1. Langford, J. (2026). The End of AI Frontiers. hunch.net, posted 6 November 2026, updated 7 November 2026.
  2. Wang, X., Cai, Z., Zhan, Z., Dong, H., Fan, Y., de Rosa, G., Pearce, T. and Langford, J. (2026). Full-bandwidth transformer. arXiv:2608.08888, submitted 9 August 2026, revised 1 October 2026.
  3. ByteDance Seed (2026). Recycler: Cost-Free Latent Reasoning via Hidden State Recycling. ByteDance Seed, 25 October 2026.
  4. Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556, submitted 29 March 2022.
  5. Nayak, A. K. and Varshney, L. R. (2025). An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models. IEEE Journal of Selected Topics in Signal Processing, 19, pp. 1338-1348, October 2025.
  6. Tiezzi, M., Casoni, M., Betti, A., Gori, M. and Melacci, S. (2026). State-space modeling in long sequence processing: a survey on recurrence in the transformer era. Neural Networks, 193, 108039, January 2026 (online 29 August 2025).