A ten-trial success rate has a 95% confidence interval roughly fifty percentage points wide. Most manipulation papers run ten to twenty-five trials per condition. The instrument this field uses to judge its generalist policies is coarser than the effects it is asked to detect.
What that instrument is measuring is good. Robot foundation models generalize, and the effect is real enough to have survived thousands of physical trials. RT-2 executed instructions on objects it had never seen. π₀.₅ ran in kitchens and bedrooms that were not in its training set. Open X-Embodiment pooled a million trajectories from twenty-one institutions and shipped it. None of that is vendor theater.
What the headlines leave out is the level all of this happens at, and how fast it decays once you chain tasks together. Cross-embodiment pretraining, meanwhile, has been measured hurting performance as readily as helping it, depending on the data regime.
Papers report gains. Gains are ratios, and ratios hide levels. RT-2 is the best-documented case because Google ran roughly 6,000 real-world trials and published the split-by-split table rather than an average.
RT-1, the smaller predecessor, tells the same story with a different shape: 97% on seen tasks, 76% on novel instructions, 83% with distractors, and 59% against unseen backgrounds — measured across more than 3,000 real trials. The pattern is consistent across the family. In-distribution, these systems are close to solved. One axis away from the training data, they are coin-flip-adjacent.
OpenVLA, the open 7B model trained on 970,000 demonstrations, averaged 70.6% on the WidowX/BridgeData V2 evaluation against RT-2-X’s 50.6%. Across 29 tasks spanning both the WidowX and Google Robot embodiments it beats RT-2-X by 16.5 points absolute — a model that still fails roughly three attempts in ten.
Single-task rates flatter these systems. Compounding error is the old, known failure of behavior cloning, and it arrives the moment you ask for more than one thing in a row.
The discrete-versus-continuous gap here interests me more than the absolute levels. Hold the backbone fixed, change only the action head, and the five-step number moves from 10.3% to 27.5% — the paper’s best configuration overall, on a different backbone, gets to 82.6%. That is an architecture result masquerading as a scaling result, the kind of thing that gets lost when a field reports averages. It rhymes with what I wrote about small models on device: the gains that survive contact with reality are usually structural, not sheer size.
The standard prescription for both of those problems is more data, and robotics has assembled a corpus to supply it. Open X-Embodiment is the field’s ImageNet moment on paper: over a million trajectories, 22 robot embodiments, 527 skills, 160,266 tasks, pooled from 60 pre-existing robot datasets contributed by 21 institutions across 34 labs. It is an extraordinary act of coordination.
It is also where the analogy to language pretraining breaks. The Open X-Embodiment paper itself reports that in its data-rich domains — Bridge, and the RT-1 dataset — RT-1-X trained on the pooled cross-embodiment corpus does not beat an RT-1 baseline trained only on its own embodiment’s data, which the authors attribute to underfitting at that model capacity. In the small-data domains the pooling worked well, beating single-embodiment baselines on four of five datasets. So the flagship dataset paper reports its own corpus helping exactly where you already lack data and failing where you don’t — a transfer story, not a scaling one.
The 2026 Nature Machine Intelligence study that systematically ablated VLA design choices found the same thing more sharply. Co-training on Open X-Embodiment without post-training did not help significantly, and post-training on it improved high-frequency skills like pick-and-place while decreasing performance on lower-frequency skills — “move near,” “open/close drawer.” The corpus is dominated by pick-and-place, so training on it is partly training the model to forget everything else. Cross-embodiment pretraining did help in the data-scarce regime: a 17.2% improvement in few-shot CALVIN. Which is roughly the opposite of the language-model story, where scale is the thing that helps most when you already have plenty.
The sharpest single data point comes from TinyVLA, published in RA-L last year: a 1.3B model fine-tuned on 100 demonstrations per task hit 94.0% average success across five real tasks over 20 trials each, against OpenVLA’s 68.3% on the same setup — with 5.5× fewer parameters and no Open X-Embodiment pretraining at all. And on bimanual tasks, OpenVLA — pretrained on a corpus containing only single-arm data — scored 0%. Every trial. That is not a small transfer penalty; that is a corpus teaching a model something actively wrong about the body it is in.
The one clean scaling result I trust points somewhere else entirely. A 2024 study collected over 40,000 demonstrations and ran more than 15,000 real-world rollouts to measure what actually scales, and found a power-law relationship with the number of environments and objects, not with raw demonstration count. Thirty-two diverse environment-object pairs at about fifty demonstrations each — four people, one afternoon — got two tasks to roughly 90% in novel settings. If that replicates, the lesson is that robot data has a diversity exponent, not a volume one, and most of the field has been optimizing the wrong axis.
Every number above came out of an instrument. Manipulation evaluations are physical, expensive, and therefore small. The Nature Machine Intelligence ablation study used three rollouts per real-world condition. TinyVLA used twenty per task for single-arm and ten for bimanual. Ten to twenty-five is the field norm.
Run the binomial arithmetic on that. (This next figure is my own calculation, not a citation — Wilson score intervals at 95%.)
Separating a genuine 70% policy from a genuine 50% one at conventional power takes on the order of ninety trials per arm — again my arithmetic, not a published figure. Almost nobody runs that. A 2025 study of repeatable robot measurement makes the same point from the metrology side: ISO 9283 recommends 20–50 samples per reference point for industrial-arm repeatability, and the authors found even that insufficient for humanoid evaluation, using 250–700 samples in response to the variability they observed.
Then there is the setup itself. The MIRRER work on reproducibility in robot learning quotes the field’s blunt working assumption — that building precisely reproducible robotic setups is impossible, and therefore absolute benchmark numbers are close to meaningless — and cites the RB2 benchmark’s observation of roughly 20% performance variation when the same experiment was reproduced across two labs. A 2025 Nature Machine Intelligence piece on reproducible robotics notes that a 2009 survey found zero of ten representative SLAM and navigation papers fully met reproducibility requisites, with a 2023 surgical-robotics study finding much the same. The remedy it endorses is not better absolute numbers but RB2’s local relative ranking: compare a method against your own lab’s baseline and stop pretending the cross-lab number means anything.
That is a reasonable adaptation. It is also an admission that the field’s headline leaderboard is not a measurement.
In 1979 Hans Moravec’s Stanford Cart crossed twenty-meter obstacle courses using stereo vision. His own write-up in Proceedings of the IEEE records the pace: the Cart “moved one meter every ten to fifteen minutes, in lurches,” stopping after each lurch to take pictures and “thought about them for a long time” on a lightly loaded DEC KL-10. Each course took about five hours. Effective velocity: three to five meters per hour.
Moravec would formulate his paradox nine years later. He had already measured it. A machine that could reason about a scene needed five hours to walk across a room — roughly a thousandth of human walking speed. The numbers in this essay are the same paradox with better hardware: the semantic part is nearly free now, and the physical part still costs everything.
So I’m watching three things. Whether the diversity-not-volume scaling result replicates outside its original lab, because it reorganizes where the money should go. Whether anyone publishes real-world manipulation results with enough trials to carry confidence intervals — and whether reviewers start asking. And whether cross-embodiment corpora get rebalanced by skill rather than aggregated by availability, since the negative transfer observed so far looks like a composition problem, not a fundamental one.
The models are better than the numbers suggest in some ways and worse in others, and right now the evaluation apparatus cannot tell us which. That is a solvable problem, and it is cheaper to solve than another million trajectories.
References