The AI Weather Models Won. Look Closely at What They Won At.

In 1908 Felix Exner made a weather forecast that worked. Assuming geostrophic balance and constant thermal forcing, he derived a simple advection equation for the pressure pattern and got a realistic result for a four-hour period over the central United States on 3 January 1895. Fourteen years later Lewis Fry Richardson published the forecast everyone remembers instead: physically complete, hand-computed from the primitive equations, and wrong by a six-hour pressure change of 145 hPa. Richardson knew about Exner’s work. He gave it one sentence, on page 43 — Exner “has published a prognostic method based on the source of air supply” — and Peter Lynch, in his history of the episode, supplies the dry gloss: “It would appear from this that Richardson was not particularly impressed by it!”

The crude method that worked got a line. The physically complete method that failed got a book, and the century.

The machine-learning weather models have now run that asymmetry in reverse. They really do beat the physics. GraphCast beat the ECMWF high-resolution forecast on 90% of 1,380 verification targets; GenCast beat ECMWF’s 50-member ensemble on 97.4% of 1,320 targets, and on 99.8% of them beyond 36 hours. ECMWF now runs two data-driven systems operationally. These are not vendor press releases — they are Science and Nature papers, and the incumbent being beaten is the same organization that went on to deploy the challenger.

So this time the crude method is winning, and it is being handed the century. The scoring rule those wins are measured with pays a premium for exactly the failure mode these models have; the models consume an atmospheric state produced by the system they are being compared against; and on ECMWF’s own scorecard the answer to “which one is better” changes depending on whether you check against the model’s own analysis or against actual thermometers. This is the same problem I keep running into with benchmarks generally: the number is real, and it is still measuring something narrower than the claim it gets used to support.

The score pays a premium for blur

Every one of these models is trained to minimize a squared-error-type loss, and most are scored with one. That combination has a known consequence that predates machine learning entirely. Minimizing mean-squared error against an uncertain future returns the conditional mean of the outcome distribution, not a sample from it — and the conditional mean of a chaotic system is smooth. The forecast that scores best is the forecast that stops committing to detail.

The verification literature has a name for the flip side. Elizabeth Ebert, in Neighborhood Verification (Weather and Forecasting, 2009), sets out the double-penalty problem: point-by-point scores “severely penalize finescale differences that are not present in coarser-resolution forecasts,” so a sharp forecast that puts a rainband 40 km off gets charged twice — a miss where the rain was, a false alarm where it wasn’t — while a smeared forecast that commits to nothing gets charged once, softly. ECMWF states the same thing in its own science blog. A model trained on that objective learns the lesson.

Massimo Bonavita, at ECMWF, measured how far the lesson goes. In On some limitations of current machine learning weather prediction models (Geophysical Research Letters, 2024) he takes Pangu-Weather apart spectrally and finds that “the effective resolution of Pangu-Weather forecasts is closer to 500-700 km than to the nominal 0.25 deg” — and that it degrades with lead time, most sharply in the first 24 hours. The IFS spectrum, by contrast, stays close to the ERA5 analysis out to about wavenumber 200. Pangu’s diverges from wavenumber 60–80.

Bonavita’s conclusion is worth quoting exactly, because it is stronger than the usual hedging: these forecasts “do not have the fidelity and physical consistency of physics-based models and their advantage in accuracy on traditional deterministic metrics of forecast skill can be at least partly attributed to these peculiarities.” The blur is not a tolerated side effect of winning. It is part of how the winning happens.

Where the blur becomes a different storm

Smoothing is harmless until the thing you need is an extreme. Two independent case studies land in the same place.

For Typhoon Doksuri, Bonavita compares a t+132h forecast: Pangu-Weather predicts a minimum central pressure of 986 hPa, a shallow low. The IFS predicts 957. IBTrACS records 944. For Storm Ciarán, Charlton-Perez and colleagues (npj Climate and Atmospheric Science, 2024) find all four ML models tested predicting maximum 10-metre winds of 25–26 m/s at 48 hours, against 34 m/s in the IFS analysis and 36 m/s in the IFS high-resolution forecast — and they specifically rule out the easy explanation, noting the weakness is not simply inherited from ERA5’s 31 km training resolution, because NWP models at comparable resolution do not show it.

The mechanism is not mysterious. Xu and colleagues (npj Climate and Atmospheric Science, 2025) find Pangu-Weather giving Doksuri a maximum surface wind of 23.9 m/s against 48.3 m/s from synthetic aperture radar, an intensity RMSE of 29.3 m/s — and then show that feeding those same Pangu large-scale fields into a 2 km WRF regional model with vortex initialization cuts the intensity RMSE to 5.1 m/s. The large-scale fields were fine. The model simply cannot put a real vortex on a grid that coarse, and the loss function never asked it to.

Change the yardstick, change the winner

The detail I found most clarifying comes from ECMWF’s own operational release note for AIFS ENS (51 members, approximately 30 km, now running alongside the 9 km IFS ensemble). ECMWF reports improvements reaching up to 25% for upper-air variables. It also reports, in the same paragraph, that “for early lead times, AIFS ENS forecasts can appear less skilful than IFS ENS forecasts when verified against IFS analyses; however, this degradation of AIFS ENS compared to IFS ENS is not visible when SYNOP and radiosonde observations are used for verification.”

The ranking flips with the yardstick. And it flips both ways: against station observations, ECMWF reports IFS ENS still more skilful than AIFS ENS for 10-metre wind speed. This is the ERA5 circularity problem made concrete — these models are trained on a reanalysis, scored against analyses from the same lineage, and the reanalysis is itself the output of the assimilation system belonging to the incumbent. Ben-Bouallègue and colleagues (BAMS, 2024) did the honest version of this test, verifying Pangu against both operational analyses and synoptic observations, and found comparable skill under both — so the circularity is not fatal. It is just never free, and it is rarely stated.

One more ECMWF self-report worth keeping: AIFS ENS “is currently overdispersive for a range of upper-air variables.” The broader ML literature mostly worries about underdispersion. The direction of the calibration error is not even settled, which is a reasonable sign that this is early.

The half nobody replaced

The efficiency claims are the least disputed part of the story and the most misread. GraphCast produces a 10-day forecast in “less than a minute on a single Google TPU v4 machine”; GenCast takes about eight minutes on a TPU v5 for a 15-day ensemble member set; ECMWF claims AIFS generates forecasts over ten times faster while using roughly a thousandth of the energy. I have no reason to doubt any of it.

But an ML forecast starts from an analysis — a physically consistent estimate of the current atmosphere, produced by assimilating millions of observations. The ML models do not do this. They are handed it, by the NWP centre they are being benchmarked against. Peter Bauer, in What if? Numerical weather prediction at the crossroads, sizes both halves. At 9 km, each ENS member uses about 50 compute nodes, so “completing the forecasts in about one hour requires therefore at least 2,550 nodes.” And the assimilation side is not a fraction of that — it is the same size: “as the EDA runs its outer-loop trajectories at the same resolution as ENS, the total node allocation is about the same as for ENS.”

So the ledger is roughly two equal halves, and the machine-learning model replaces one of them. Bauer adds the context that makes this bite: the two operational clusters comprise 3,840 nodes, and the assimilation, forecast and boundary-condition suites together “already claim 80-85% of the available capability.” The “1,000× less energy” claim is true of the part being replaced and silent about the part being inherited. That is a real and large saving. It is not the end of the supercomputer, and the people at ECMWF have never said it was — Bauer’s whole paper is an argument for using ML to relieve a machine that is running out of room.

The operational posture reflects this. NOAA’s National Hurricane Center describes integrating AI systems “as guidance when preparing operational forecasts, alongside all of the other critical tools in our toolbox,” while noting that verification over many storms shows “the official forecast is the most skillful and consistent, surpassing any individual model forecast.” Nobody is issuing warnings off a neural network alone.

Richardson’s actual mistake

Which brings me back to 1922. Richardson’s instantaneous pressure tendency was about 0.7 Pa/s, which is not absurd at all — Loehrer and Johnson later measured −1.3 Pa/s in a real mesoscale convective system. The catastrophe came from multiplying a noisy instantaneous value, contaminated by unfiltered gravity waves, by a six-hour timestep. His physics was fine. What he had was a number that meant something much narrower than the use he put it to.

That is the failure mode I would watch for now. Three things are true at once, and the press coverage tends to carry only the first. The data-driven models beat the physics on published scores. Those scores reward the smoothing that makes them dangerous precisely where forecasts matter most — the deep low, the intensifying cyclone. And they are one half of a pipeline whose other half is still a supercomputer solving equations, which is also the half that decides what “the current state of the atmosphere” even means.

That is not a story about AI replacing simulation. It is closer to what I argued about digital twins: the learned model earns its place by being fast enough to run a thousand times, on top of a physical model that stays responsible for being right. I would not want to forecast without these models. But if you are reading a forecast scorecard, ask what the verification target was, and whether anyone checked against a thermometer.

---

References

1. Lam et al. (2023). Science. arXiv:2212.12794.
2. Bi et al. (2023). Nature. arXiv:2211.02556.
3. Pathak et al. arXiv:2202.11214.
4. Price et al. (2024). Nature. arXiv:2312.15796.
5. Bonavita, M. (2024). On some limitations of current machine learning weather prediction models. Geophysical Research Letters.
6. Charlton-Perez et al. (2024). npj Climate and Atmospheric Science.
7. Xu et al. (2025). npj Climate and Atmospheric Science.
8. Ben-Bouallègue et al. (2024). BAMS. arXiv:2307.10128.
9. Bauer, P. What if? Numerical weather prediction at the crossroads. arXiv:2407.03787.
10. Ebert, E. (2009). Neighborhood Verification. Weather and Forecasting.
11. Bracco et al. (2024). Nature Reviews Physics.
12. Eyring et al. (2024). Nature Climate Change.
13. Emanuel. (2023). Nature Climate Change.
14. ECMWF Newsletter 183.
15. ECMWF. Newsletter 185: AIFS ENS becomes operational.
16. NOAA/NHC. AI Hurricane Forecasting. weather.gov.
17. Lynch, P. History of the episode, from Weather Prediction by Numerical Process, ch. 1.