On 3 September 2026 the ARC Prize Foundation published its evaluation of OpenAI’s GPT-6 Astra on ARC-AGI-3, the interactive benchmark it launched in March. Two numbers are in the summary. Under the Foundation’s Standard harness, Astra at maximum reasoning effort scores 62.7% on the Semi-Private set for $26,098. Under a Provider Adapter harness, which lets OpenAI’s own context-management features run, Astra at high effort scores 99.9% for $18,817.
OpenAI’s launch materials lead with the second number. Most of the coverage I have seen leads with it too. I think the first number, and the distance between the two, is the more informative result, and I want to explain why before getting to the part of the report that surprised me.
ARC-AGI-3 is not a question-answer benchmark. Each task is a small turn-based game on a grid with no instructions, no stated goal, and no text. The agent has to press things, see what happens, work out the rules, work out what winning means, and then win, level after level. The Foundation’s technical report lists 25 public demonstration environments, 55 semi-private environments used to test models behind an API, and 55 fully private environments held for the competition. When it launched on 25 March 2026, humans solved 100% of the environments and frontier models scored under 1%.
The scoring is the unusual part. ARC-AGI-3 does not just count completed levels. It scores action efficiency: for each level, how many moves the agent took compared with a human baseline, which the Astra post defines as the median action count among the human testers who completed that level. The baseline came from in-person sessions with roughly 500 members of the public who were not selected for puzzle skill. Per-level efficiency is capped at 1.15 times the human figure so that a lucky two-move exploit cannot swamp an environment’s score, and the report says the efficiency term is squared before aggregation. A perfect score means the agent beat every game at least as efficiently as a representative person seeing it for the first time.
Before Astra, the highest score the Foundation itself had verified was Claude Opus 5 at 30.2%, published on 24 July 2026 at high reasoning effort. GPT-5.6 Sol, the model Astra replaces, sits at 7.78% at max effort on its results page.
The Foundation’s testing policy now defines two evaluation conditions. The Standard harness is a minimal, provider-neutral interface: the model sees the game, takes actions, and carries forward whatever notes it chooses to write for itself. The Provider Adapter harness lets a lab’s own context-management features run. For OpenAI that means preserving the model’s opaque reasoning state between requests, which the Foundation cannot inspect, and compacting long conversations so earlier work stays available.
The post gives the full table, all six reasoning-effort levels under both conditions.
Under the Provider Adapter, reasoning effort barely matters: the spread from “none” to “high” is 3.2 points. Under the Standard harness it is the whole story, with a 45-point spread and a climb from low to max, broken only by the “none” setting, which oddly beats “low”. And the gap between conditions at matched effort runs from roughly 36 to 80 points, on the same model and the same games, with the only change being whether the model’s hidden reasoning state survives from one turn to the next.
The Foundation puts the aggregate difference plainly: across the Public and Semi-Private sets and all effort levels, Provider Adapter runs were about 3.66 times faster by elapsed time and used 49% fewer tokens on the 167 game-and-effort pairs both harnesses solved. A model that has to reconstruct its understanding of a game from its own written notes each turn is a different, and much weaker, agent than one that can carry its working state forward intact.
I do not think this is a trick. The Foundation says it will report both conditions going forward, clearly labeled, and I think that is the right call. But it means “GPT-6 Astra scores 99.9% on ARC-AGI-3” is a claim about a model plus OpenAI’s context-management stack, and “62.7%” is the closest thing on offer to a claim about the model on its own. The 37-point difference is a measurement of how much of the capability lives in the harness. That is a large number, and it matches what NVIDIA reported on 21 August from a different direction: wrapping Claude Opus 5 in its AVO agent system, with persistent memory and a supervisor loop, took the model from the Foundation’s 30% to a perfect score on the 25 public environments. NVIDIA was careful to say that result is on the public set only and is not a controlled ablation. Same lesson, though. Evaluating a model and evaluating an agent are different activities, and the number you get depends on which one you did.
There is a second wrinkle in how the result was announced. OpenAI’s own materials, as reported by The New Stack, cite 98.6% via its Responses API with two settings changed to reflect production use. The Foundation’s post shows 98.6% at max effort and 99.9% at high. These are consistent with each other, but they are not the same number, and the 99.9% figure is a single run at one effort level with no averaging. The Foundation’s policy is one run, published with replays. I would treat the Provider Adapter result as “saturated” rather than as 99.9% specifically.
The cost column is the part I would have bet wrong on. Under the Standard harness, the most expensive reasoning setting is the cheapest run.
The Foundation’s explanation is simple once you see it. On an interactive task, total cost is the number of model calls times the cost per call. A model that thinks harder per move needs fewer moves, and the saving on calls outweighs the extra tokens per call. So on ARC-AGI-3, unlike on a static exam, turning reasoning effort up can make the run cheaper. That has an obvious implication for anyone budgeting agent deployments: the instinct to throttle reasoning to save money may cost you more, because the agent will wander.
The human comparison is worth stating carefully because the post does. Test participants were paid $115 for a 90-minute session plus $5 per completed game, attempting about nine games a session, or roughly $12.78 per attempted game before the completion bonus. By my arithmetic, 55 games at that rate is about $700, or under $1,000 with every bonus paid, against $26,098 for Astra’s best Standard run. The Foundation also offers the number I find least useful: if you price only the brain’s metabolic energy at 20 watts and $0.20 per kilowatt-hour, a game costs about 0.067 cents. It is a fun footnote, and nobody budgeting an evaluation would use it.
The finding I expect to matter longest is about moves, not scores. When the Foundation designed ARC-AGI-3, it hypothesized that action efficiency would stay a dividing line between humans and AI: that even when a model solved a game, it would need far more exploration to get there. That was a reasonable prior. Brute-force search agents behave exactly that way.
Astra does not. Under the Provider Adapter at max effort, it used fewer actions than the human baseline on 96.0% of levels, and 51.7% fewer actions per level on average. The Foundation calls this human parity on its own measure and I think that is a fair description. Its observation about the shape of the result is the more interesting part: frontier models now show a binary pattern. Either they have not understood the mechanics, in which case they flail, or they have, in which case they execute inside the range of human efficiency. The long middle, where a model half-understands and grinds its way through, seems to be disappearing.
The replays show how. Astra chooses what notes to carry forward between
turns, and what it wrote was a compact symbolic model of each game. The
post gives examples of the shorthand it invented. A line like
L8: hub q2 (8↓). Lengths: 14=1… records the level, a rotation index, and
mechanism lengths. extend8 to3; retract10 to2; shorten8 to1 is an ordered
plan over two colour-coded mechanisms. 9−=(39,4), rotate=(49,18),
14+=(59,11) maps operations to the screen coordinates of the controls that
perform them. The Foundation says it has seen similar behaviour in other
models, but that Astra’s notation was unusually dense and precise: closer
to an algebra it made up on the spot than to prose notes. In a separate
red-team harness with a code sandbox, it went further and wrote small
game-specific libraries, with files like maze_solver.py,
patrol_solver.py and sync_state.py for a maze game with moving guards.
The Foundation is careful to label those runs as model-plus-tools, since
the human testers had no sandbox. But the Standard-harness notation needed
no tools at all. The model wrote itself a domain-specific language because
that was the cheapest way to remember what it knew.
The Foundation’s own framing is measured, and I will borrow it. Astra clears the bar ARC-AGI-3 set: it synthesizes causal world models of unfamiliar environments and reaches goals without instructions, at or above human action efficiency. The Foundation calls it a step-function change. It also says, as it said at launch, that saturating ARC-AGI-3 is not proof of AGI, and it is not claiming Astra is AGI. ARC-AGI-3’s environments are deterministic, closed-ended and tightly formatted. They do not test open-endedness, long-horizon work in a messy world, or the capacity to improve oneself, and the Foundation says the next generation of benchmarks will be aimed at exactly those.
Three caveats of my own. The Semi-Private set is tested through a public API under zero-retention agreements, and the Foundation’s policy acknowledges a small leakage risk over time; the fully private set, used for the competition, has not been reported for Astra. The 99.9% is one run at one setting. And the 62.7% is what you get when the model has to rely on its own written memory, which is also the condition the Foundation says a future AGI should be able to pass. The distance between those two numbers is the most useful thing in the report. It says the model is very good, and it says that a large share of what makes it good on this test is machinery around the model that keeps its thinking from evaporating between turns. Both halves matter. When I wrote about how progress toward AGI gets measured, the recurring failure was an instrument being asked to carry a claim larger than the thing it measures. This is the first time I have seen the instrument’s authors publish the second number so that it cannot.
Disclosure on a source: The New Stack, whose reporting on OpenAI’s figures I cite, notes that its owner Insight Partners is an investor in OpenAI.
References
All Astra scores and costs are the ARC Prize Foundation’s figures from single runs at retail API pricing. The $700 human-cost comparison is my own arithmetic from the Foundation’s per-game rate.