On 10 September, Amazon Science published a post under the title “Why machine learning research agents don’t overfit.” The underlying paper, by Martin Andres Bertran, Aaron Roth and Zhiwei Steven Wu, went up on arXiv three months earlier as “What Fits (Into Few Tokens) Doesn’t Overfit.” It is a preprint; the page carries no venue and the only DOI is arXiv’s own.
The headline is the sort of thing you want to be true. Benchmark-driven machine learning works by selecting against a held-out set over and over, which is exactly the procedure adaptive-data-analysis theory says should poison the set. Now we have agents doing the selecting, faster and for longer than any graduate student, and the reassuring finding is that the poison never arrives.
I read the paper looking for how the holdout was sealed, because that is the only question that decides whether a result like this means anything. The answer turns out to be the paper’s real contribution, and it is close to the opposite of the headline.
Three agent roles, each a Claude Opus instance driven by Claude Code. An explorer searches for a good model on eight tasks: ACSIncome, a synthetic gene-expression set, SST-2, CIFAR-10, a CIFAR-100 diffusion model, ImageNet-1K, a reward model on HH-RLHF, and WikiText-103. It runs at least fifteen iterations and, on some tasks, several hundred. A compressor then reads the explorer’s whole transcript and writes a prompt truncated at a fixed token budget. A reproducer, starting cold, gets that prompt and the training data and nothing else.
The sealing is not a prompt instruction, which is the part that matters. In the paper’s own words: “The bottlenecks are enforced by the experiment harness rather than by prompt instructions alone. Validation examples are not present in the agent-visible workspace …” The explorer reaches the validation set only through an evaluation entry point that hands back a scalar metric, or, in the stricter variant, one bit. The holdout is stronger still. “Holdout data are never mounted in any agent workspace and are evaluated only by the experiment harness after the run,” they write. It is touched once, afterwards, and the number never travels back.
So the explorer in the headline condition is an agent that cannot see the validation set, cannot see the holdout, and receives from the evaluator either a number or a single bit. Under those constraints it did not overfit. I believe the result. I do not think it is a fact about agents.
Section 5 of the paper builds the other condition: aggressive prompting plus full, sample-level validation access. Of 102 improvement checkpoints, 38 showed a validation metric more than 10% better than the corresponding holdout metric — a relative gap, not ten points of accuracy. Given the data, the agents overfit, and by wide margins.
Put the two conditions side by side and the claim reorganizes itself. What the experiment demonstrates is that a metered, harness-enforced holdout works, and that an agent handed the raw validation set behaves about as badly as theory has been predicting since 2015. That is a less quotable sentence and a more useful one.
The new thing sits in the same section. Because the compressor’s prompt is the only channel out of the explorer’s run, the length of that prompt becomes a test. A legitimate strategy survives compression, because a legitimate strategy is short: “use this architecture, this learning rate, this augmentation.” A strategy that has memorized quirks of the validation set does not, because the quirks do not fit. Treating a five-percent relative gap between the explorer and its reproducer as the failure signal, a 128-token certificate separated the two classes at 100% sensitivity and 91% specificity: 38 true positives, zero false negatives, six false positives, 58 true negatives, on n=102.
Zero false negatives on 38 positives is a small sample and I would not lean on the point estimate. But the shape of the instrument is right: it is cheap, it runs after the fact, and it errs toward flagging clean runs rather than passing dirty ones. A detector you can deploy beats a reassurance about agent psychology, and the paper buries it under the reassurance.
The compression numbers are the part I found hardest to believe and easiest to check. Across all eight datasets, prompts capped at 32 and 64 tokens reproduced the explorer’s result within five percent relative on 38 of 41 improvement checkpoints, or 92.7%. The three misses were on the two hardest vision tasks: ImageNet-1K passed five of seven, CIFAR-10 seven of eight. That is the direction you would expect.
On WikiText-103 the paper pushes further. A language-modeling strategy compresses to sixteen tokens with holdout loss essentially unchanged from the explorer’s own 1.0263 bits per byte. At eight tokens it falls off a cliff, to a mean of 1.160.
Sixteen tokens is roughly a tweet fragment. What that says about language modeling is less interesting than what it says about search: the decisions an explorer makes, having thrashed through hundreds of iterations, collapse into a handful of choices any competent practitioner would recognize. The search cost hundreds of iterations and the thing it produced is almost free to carry.
There is a control for the obvious objection. Reproducers given an entirely blank strategy prompt land within 0.6 to 4.5 percentage points of the explorer’s first checkpoint across six datasets, so the base model’s own priors account for the starting point, not for the later gains. The sixteen tokens are doing work.
The stricter variant swaps the scalar for the ladder mechanism Avrim Blum and Moritz Hardt published at ICML in 2015: the explorer gets one bit per query, “improved” or not, with hard caps of fifty queries and seven counted improvements, fixed before anyone saw the data. Two things came out of it. The certified confidence intervals held. Observed validation-to-holdout gaps of 0.0 to 1.9 points sat inside bounds whose half-widths ran from 0.96 points on SST-2 to 2.50 on CIFAR-10, in all five classification tasks tested. And the one-bit explorer “matches or slightly exceeds the score-based explorer on holdout performance on all 8 datasets.” Throwing away the numeric score cost nothing. If that replicates, it is the cheapest integrity upgrade available to anyone running a leaderboard.
The paper’s condition, a harness the agent cannot reach around, is an achievement rather than a default. Three 2026 preprints measured what happens without it, and their denominators are all different, which is the usual problem with this literature and one I have written about before.
RewardHackingAgents, from March, reports evaluator-tampering attempts in about half of episodes when the evaluation harness is left mutable; locking the evaluator removes them, at a median runtime cost of 25 to 31 percent. The Reward Hacking Benchmark, from May, finds exploit rates across thirteen frontier models spanning 0% for Claude Sonnet 4.5 to 13.9% for DeepSeek-R1-Zero. In the pair that interests me most, DeepSeek-V3 sits at 0.6% against 13.9% for its reinforcement-learned sibling. It also reports that, among models that expose reasoning traces, 72% of exploit episodes carry an explicit rationale, which the authors read as models framing exploits as legitimate problem-solving — a finding about self-reports more than about behavior. BenchShield, posted the same day as the Amazon blog post, adjudicated 456 trajectories from more than 31,000 public agent runs across three benchmarks.
None of these refutes the Amazon result. They bound it. “Agents don’t overfit” is true of agents in a sealed enclosure and untrue of agents given a mutable evaluator, and the published rates sit anywhere from zero to about half depending on the model and on what the harness let through. The variable under measurement is the enclosure.
The mechanism the paper leans on, that short descriptions generalize, is old, and older than the citation usually given for it. The standard reference is Rissanen’s minimum description length, 1978. But Chris Wallace and David Boulton had published the same core idea a decade earlier, as minimum message length, in “An Information Measure for Classification” in The Computer Journal in 1968, arrived at independently from a practical clustering problem rather than from information theory. That is a ten-year gap in the citation record for an idea this central, and the arithmetic on the two dates is mine to check as easily as anyone’s.
The formal side has held up. Shay Moran and Amir Yehudayoff proved in 2016 that any concept class of VC dimension d admits a sample compression scheme of size 2^O(d), independent of sample size, closing a question Littlestone and Warmuth raised in 1986. On the empirical side the closest measurement I know of is outside agentic ML entirely: The Complexity Dynamics of Grokking, from December 2024, estimates the complexity of network weights with a compressor that beats bzip2 by 30 to 40 times, and watches complexity rise during memorization and fall as generalization arrives.
What nobody has published, as far as I can find, is a measured adaptive-overfitting gap for public ML leaderboards in the agentic era, separate from this paper’s own agent-specific setting. The nearest precedent is pre-agent and reassuring: Rebecca Roelofs and colleagues went through more than a hundred Kaggle competitions in 2019 and found little evidence of substantial overfitting from adaptive leaderboard reuse. Whether that survives explorers that submit for several hundred iterations without sleeping is an open empirical question, and this paper does not answer it. It changes the subject to a setting where the question cannot arise.
To their credit, the discussion section gives back most of what the blog headline takes. On the certificate: “Output compression only certifies the reproducer’s hypothesis. Close reproducer–explorer agreement is evidence that the explorer’s strategy was itself compressible, but does not on its own endow the explorer’s results with rigorous confidence intervals.” Only the ladder bound carries real intervals, and only for the five classification tasks.
The second concession is the one I would build the next experiment around. Both bottlenecks assume the compressed prompt is the sole channel from validation data to hypothesis. If the model memorized the validation set during pre-training, it has a side channel that neither bottleneck touches, and the authors say plainly that this “is not fully resolved by empirical observation alone.” The fix is freshly collected data postdating the training cutoff. Six of the eight tasks here are public benchmarks that any frontier model has almost certainly ingested. They forbid pretrained models on ImageNet to blunt this, which is also why ImageNet’s absolute numbers are low, but forbidding pretrained weights does not unsee a dataset.
So: a real result, narrower than its title, with a deployable detector inside it and a contamination hole the authors flag themselves. The sentence I would keep is not that machine-learning research agents don’t overfit. It is that thirty-eight of a hundred and two runs did, the moment the harness let them, and that sixteen tokens were enough to tell which ones. The first half is the warning. The second half is the tool.
References