By the ninth generation, an article that began on Somerset church architecture had wandered into a list of “black-tailed jackrabbits, white-tailed jackrabbits, blue-tailed jackrabbits, red-tailed jackrabbits, yellow-…” — grammatical, confident, and completely unmoored. That is the most-quoted output in the model-collapse literature, and it is real: Shumailov et al. produced it by training each generation of a language model on the previous generation’s output. It is also, I have come to think, the most misread result in the field.
The misreading is treating collapse as a property of synthetic data. It is a property of a pipeline — one in which machine output replaces human data rather than adding to it. Change that one assumption and the theory changes with it: accumulate instead of replace and the error is provably bounded; attach a verifier and synthetic data beats human data on the benchmarks its authors chose. What degrades under essentially every regime, including the ones that work, is diversity — the part of the story I think gets underplayed, with the least reassuring literature behind it.
The reference result is Shumailov et al., Nature 631:755–759, published 24 July 2024. They define model collapse as a degenerative process in which generated data pollutes the training set of the next generation, and they split it into two phases: early collapse, where the model “begins losing information about the tails of the distribution,” and late collapse, where it converges to something “with very small variance” bearing little resemblance to the original.
The language-model experiment behind the jackrabbits is small and clean. They fine-tuned OPT-125M on WikiText-2, generated text from it with five-way beam search, fine-tuned the next generation on that text, and repeated. Nine rounds of that produced the list above.
The theory is stronger than the demo. For discrete distributions under perfect function approximation, any state with probability q has probability roughly 1−q of not appearing in a finite resample, so the process is a Markov chain whose only absorbing states are delta functions: the model converges almost surely to a point mass. For the multidimensional Gaussian case, the expected Wasserstein-2 distance to the true distribution diverges with generations and sample variance goes to zero with probability one. They attribute this to three compounding sources — statistical approximation error from finite resampling, functional expressivity error from limited capacity, and functional approximation error from the training procedure — of which the first is the primary source, the one that remains even under perfect functional approximation.
Alemohammad et al. (ICLR 2024) reached a compatible conclusion for image generators: without enough fresh real data each generation, an autophagous loop is “doomed to have their quality (precision) or diversity (recall) progressively decrease.”
Read the setup again. In the primary condition, each generation trains on the previous generation’s output with no original data preserved. A softer variant keeps a random 10% of the real data every round; it degrades more slowly and still degrades. Neither is what a competent lab actually does, which is keep everything and add to it.
Gerstgrasser et al. asked exactly that question in “Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data” (COLM 2024, arXiv:2404.01413). Their result, in an analytically tractable sequence of linear models: if data are replaced, test error increases with the number of model-fitting iterations — reproducing the collapse result — but if data instead accumulate, “the test error has a finite upper bound independent of the number of iterations.” Unbounded growth becomes a ceiling. They report the same contrast empirically across language models, diffusion models for molecular conformation, and VAEs.
There is a third regime, and it is the one people conflate with the other two. Dohmatob et al., “Strong Model Collapse” (ICLR 2025), show that in a growing corpus that carries a fixed synthetic fraction, “even the smallest fraction of synthetic data (e.g., as little as 1% of the total training dataset) can still lead to model collapse” — larger training sets stop helping. That is not a contradiction of Gerstgrasser. It is a different experiment: a fixed contamination share versus an unboundedly growing accumulated pool. Which one describes your situation depends on whether you control your corpus or scrape it.
The success stories are not marginal, and they are not vendor slideware.
AlphaGeometry (Trinh et al., Nature 625:476–482, 18 January 2024) trained on roughly a billion randomly generated geometric diagrams, symbolically deduced down to 100 million unique synthetic theorem-and-proof examples, about nine million of which required auxiliary constructions. Zero human demonstrations. It solved 25 of 30 benchmark olympiad geometry problems inside competition time limits, against 10 of 30 for the previous best symbolic method.
Microsoft’s phi-1 (arXiv:2306.11644) is a 1.3B-parameter model trained on about seven billion tokens — roughly six billion of filtered “textbook-quality” web text plus about one billion of GPT-3.5-generated synthetic textbooks and exercises — reaching 50.6% pass@1 on HumanEval and 55.5% on MBPP, on eight A100s in four days.
Self-Instruct (Wang et al., ACL 2023, pp. 13484–13508) bootstrapped 52,445 instructions and 82,439 instances from 175 seed tasks, rejecting any new instruction with ROUGE-L overlap above 0.7 against existing ones, and reported a 33% absolute improvement over vanilla GPT-3 on SUPER-NATURALINSTRUCTIONS — on par with the human-tuned InstructGPT-001.
And Meta’s Llama 3 paper (Grattafiori et al., arXiv:2407.21783) says the quiet part plainly about post-training: “most of our training data is model-generated.” They describe roughly one million synthetic coding dialogues built through an execution-feedback loop plus about 1.2 million more via backtranslation.
The common factor is not synthesis. It is selection against error. AlphaGeometry’s oracle is a symbolic deduction engine. Llama 3’s is a test runner. Phi’s is a curation filter. In “Think, Prune, Train” (Costello et al., ICLAD 2025), recursive fine-tuning on a model’s own reasoning traces pruned by ground-truth correctness lifted GSM8K pass@1 from 41.9% to 57.6% on Gemma2-2B and from 78.6% to 91.5% on Llama-3.1-70B. Run the identical loop without the correctness filter and the result does not simply invert. Their no-pruning ablation leaves GSM8K pass@1 slightly above baseline, at 45.1%, while pulling pass@20 below it — 71.6% against 76.0%. The filter is not what makes self-training work at all. It is what stops the model from paying for its first-guess accuracy with its breadth.
None of this is new, either. The earliest formalization of training a classifier on its own labels is H. Scudder’s 1965 paper, “Probability of error of some adaptive pattern-recognition machines,” IEEE Transactions on Information Theory 11(3):363–371 — self-training had published convergence conditions sixty-one years ago, and the conditions were the interesting part. They require an initial classifier good enough and unbiased enough that its self-generated labels do not systematically drift, which is precisely the assumption indiscriminate LLM self-training violates. The 2020s did not discover the technique. They discovered what happens when you run it outside its stated preconditions.
Here is the caveat I would not skip. Even the pipelines that work trade away diversity, and diversity is harder to see on a dashboard than accuracy.
Grigoreva, Stinson and Muise (FLLM 2025) measured 3-gram diversity — distinct trigrams over total trigrams — across self-consuming generations of a GPT-2-class model. Their paper is paywalled and I could not check the figures below against an open preprint, so read them as indicative shape rather than settled values. On haiku it fell from about 0.6 to about 0.1 in the first generation; on tweets from about 0.7 to about 0.2; on the more complex Yelp corpus from about 0.6 to about 0.1 over ten generations, plateauing in the 0.1–0.2 band by the fourth. Their quality metrics fell monotonically alongside, with the share of logically valid statements settling near a 10% floor.
The narrowing shows up in the successful pipelines too, just politely. The Think-Prune-Train authors report pass@1 climbing steadily while pass@20 and pass@50 plateau or slip. The model is concentrating probability mass on the answers it is already confident about. For math and code that is close to free — you wanted the first sample to be right. For open-ended generation, and for anything where you were planning to spend a search budget on sampling widely, you have quietly sold the thing you were going to spend. It is the same measurement problem I ran into writing about evaluation in robot foundation models: the headline metric improves while the metric nobody reports is what actually moved.
Less measurably than the discourse implies. There is no peer-reviewed census of what fraction of the web is machine-generated. The defensible numbers are corpus-specific and use different units.
Liang et al. (Nature Human Behaviour 2025;9(12):2599–2609) applied a population-level distributional estimator to more than 1.1 million papers and put the share of LLM-modified sentences in arXiv computer-science abstracts at 22.5% by September 2024 (95% CI 21.7–23.3%), against 7.7% for mathematics and 8.9% for Nature-portfolio journals. Their pre-ChatGPT baseline of roughly 2–3% is the method’s own false-positive floor, and they note it overestimates at the low end and underestimates at the high end.
Kobak et al. (Science Advances) took a different route — excess vocabulary, tracking the post-2023 jump in words like “delve” against their own historical baselines — across more than 15 million PubMed abstracts, and concluded that “at least 13.5% of 2024 abstracts were processed with LLMs,” reaching 40% in some subcorpora. They frame it explicitly as a lower bound, and it is: the method cannot see LLM text that avoids the tell-tale vocabulary.
The detection literature is where the confidence should drain out. A 2025 review in Computer Science Review — also paywalled, so the figures in this paragraph are as I found them reported rather than as I read them in the primary text — documents that paraphrasing and light human editing cut detector accuracy by roughly 40–60%, that locating the boundary where human text becomes machine text tops out near 23% exact match, and that detectors flag non-native English writers at up to about 2.5× the false-positive rate. Any global percentage you see quoted is resting on that machinery.
If I were running one of these pipelines, in decreasing order of confidence:
Keep the real data and never replace it. This is nearly free and it moves you from the panel with unbounded error into the panel with a ceiling.
Buy or build the verifier before you buy the synthetic data. In every result above where synthetic data won, something outside the model was checking the answers — an executor, a symbolic engine, a labeled test set. Synthetic data without a verifier is the experiment that produced the jackrabbits.
Measure diversity, not just accuracy. Distinct-n, self-BLEU, pass@k at several k. If pass@1 is rising while pass@50 is flat, you are trading breadth for confidence, and you should at least know you agreed to it. This is the same discipline I keep arguing for in measuring progress toward AGI — pick the instrument before you pick the number.
And treat every “X% of the web is AI-generated” figure as a lower bound with a wide, biased error bar attached to a corpus that is probably not yours. The honest version of that sentence is that nobody has measured it.
References