In 2024, frontier models solved about 15% of the challenges in Cybench, a capture-the-flag benchmark. In 2025 they solved 93%. That jump is from Stanford HAIβs 2026 AI Index, published in April, and it is the unguided figure, with no hints and no decomposition into subtasks. Anthropicβs system card for Claude Mythos Preview reports a pass@1 of 100%, though on a 35-challenge subset of the 40 rather than the whole set, and at ten trials per challenge where the comparison models ran thirty. That is a vendorβs own measurement, and a model evaluated at ten trials against comparisons at thirty is not quite a like-for-like table. On the conclusion, though, Anthropic agrees with its critics: the card says Cybench is saturated and βno longer sufficiently informative of current frontier model capabilities.β Cybench is finished.
Finished at what, though. I spent this morning going through what the 2026 literature actually measures about autonomous cyber agents, and the thing that stopped me was not a capability result. It was a column in a table that almost nobody cites.
Maria Rigaki, Carlos Catania and Sebastian Garcia built Hackphyr, a fine-tuned 7B model acting as an autonomous attacker inside NetSecGame, a network penetration simulation. The journal version of the work appeared in Expert Systems with Applications in March 2026 (DOI 10.1016/j.eswa.2025.129987); the results tables I am quoting come from the openly available preprint (arXiv:2409.11276), which is dated September 2024. Hold onto that date. They ran three scenarios of increasing size (a small two-subnet exfiltration, a larger one with five clients, and a three-network layout with firewall constraints) and reported win rates for each.
The headline numbers are good. GPT-4 wins every single episode of the small and full scenarios and 82.35% of the three-network layout. Hackphyr takes 94%, 89.1% and 50.34%, which for a 7B model running on one GPU card would have been the story anywhere else.
Then they ran the whole thing again with a stochastic defender process in the environment. It is a script that sometimes notices and responds, nothing resembling a SOC team. GPT-4βs win rates went to 83.33%, 53.33% and 36.36%. Hackphyrβs went to 59.77%, 44% and 23.33%.
Read the frontier modelβs row again. In the full scenario GPT-4 goes from winning every episode to winning barely half of them. On three networks it falls from 82% to 36%. This is not a small-model artifact you can wave away: the strongest agent in the study loses about half its wins the moment anything in the environment responds. And a scripted defender is the weakest possible model of a defended network: no threat intel, no patching cadence, no analyst who finds the whole thing weird and picks up the phone.
This licenses less than it might look like it does. NetSecGame is one simulation, GPT-4 is now several model generations old, and a scripted defender in a simulated network is not a security team. You cannot take 36% and staple it onto a frontier modelβs Cybench score. What you can say is narrower and, I think, worse: when someone bothered to measure the same agents against an environment that resists, roughly half the wins went away. And the benchmarks everyone quotes do not measure that at all.
Now the date. Those tables were published in September 2024. In the two years since, the field has produced saturated CTF scores, a 15-to-93 jump, and a great deal of argument about what frontier agents can do to a network, and as far as I can find, nobody has rerun the defender experiment on a current model. The single most decision-relevant variable in the whole area was measured once, on GPT-4, and left there.
They were built to be scoreable, which is not the same as built to be realistic. A capture-the-flag challenge has a flag. It is deterministic, it terminates, and it grades itself. A defended production network has none of those properties, which is exactly why it is hard.
The 2026 surveys keep arriving at this from different directions. A critical survey of LLMs for cyberattack defense in Knowledge and Information Systems in March (DOI 10.1007/s10115-026-02736-y) finds that most evaluation rests on offline curated datasets that miss concept drift, alert imbalance, encrypted traffic, and adversarial adaptation, which between them define the job. A systematic review in Applied Intelligence in July (DOI 10.1007/s10489-026-07340-9) reports that shared, reproducible agent benchmarks are largely absent, and that most demonstrated systems stay inside small controlled scenarios.
Two 2026 preprints put numbers on the ceiling. SRE-Bench, released in August, builds a contamination-controlled binary reverse-engineering benchmark of 1,572 tasks; the strongest model scores 61.4% per instance but fully solves only 31.5% of tasks, and the authors find that source-level security skill does not transfer to binaries (arXiv:2608.11469). A second-look study in April found that a general-purpose coding agent matched purpose-built CTF architectures at 19 of 30 tasks, and that agents and humans fail on the same challenge categories (arXiv:2605.21497). Shared failure modes point at a capability ceiling rather than a scaffolding problem.
Epoch AI made the sharpest version of the argument in June, reviewing the Mythos cyber claims specifically. Their objections: the benchmarks were near saturation before the comparison was drawn, so ceiling effects flatter the delta; discovery and exploit development were being reported as one capability when they are two; and Project Glasswing reportedly consumed up to $100 million in API credits, which means a rise in raw vulnerability counts may be a spending result rather than a capability result. This is the same shape of problem I ran into writing about what the robotaxi crash data can and cannot support: the numerator gets published and the denominator stays private.
What came out the other end is a better record than I expected.
CyberGym, whose final version landed in March 2026, is the strongest academic evidence I found (arXiv:2506.02548). Its benchmark half is sobering: given a vulnerability description and asked to reproduce the bug, the best agent combinations reach only about 20β22% on 1,507 real vulnerabilities across 188 projects. But the agents also found 34 previously unknown vulnerabilities: 25 from an open-ended run across 431 projects, and 9 more from re-testing the benchmarkβs own proof-of-concept exploits against current code. As of writing, 4 had received CVE assignments and 10 had been patched.
The DARPA AI Cyber Challenge systems have a longer trail, because the teams kept going after the August 2025 finals. OpenSSFβs May 2026 retrospective reports that FuzzingBrain found 62 vulnerabilities across 26 projects, of which 43 were confirmed by maintainers and 36 patched; Team Atlantaβs OSS-CRS found 25 across 16 projects, with 9 fixed and 8 more confirmed with fixes in progress; 42-b3yond-6ug found 12 Linux kernel vulnerabilities and 10 further zero-days in userspace projects. Two of those tools now live under the Linux Foundation.
Notice which number is doing the work. β62 foundβ is a press release; β43 confirmed, 36 patchedβ is a result. A finding that no maintainer accepts is not a finding, and the ratio between those two columns is the only part of the pipeline that costs a human being anything. This is the same distinction I ended up drawing about the four different claims people were making about one agent incident: the word does the persuading, and the evidence bar sits somewhere else entirely.
One figure sits further down in CyberGymβs evaluation. Those nine re-tested zero-days had been sitting in actively used open-source code for an average of 969 days, nearly three years, before an agent working from a template found them. And the same run turned up 18 vulnerabilities across 15 projects where a human-written patch had already shipped and had not actually closed the hole. Those were not new bugs. They were old bugs that everyone, including the maintainer who wrote the fix and the reviewer who approved it, believed were dead.
The other 2026 story is the one that reversed.
In January, Daniel Stenberg shut down curlβs bug bounty. Over the programβs life, from April 2019, curl had confirmed 87 vulnerabilities and paid out more than $100,000. Historically over 15% of submissions were real. Under the flood of AI-generated reports that rate fell below 5%. The Register, reporting the closure, noted that in the week before Stenbergβs announcement the scheme generated seven submissions, and that while some identified bugs, none described a vulnerability. The program closed on 31 January.
That is where most coverage stopped. curl rejoined HackerOne in March without monetary rewards, and by April the confirmed-vulnerability rate was back to 15β16% with report volume doubled against the 2025 peak. Stenbergβs words: βthe slop situation is not a problem anymore.β He projected curl might publish around 50 vulnerabilities across 2026, which would be a record.
Nothing in curlβs triage stack changed to produce that. Removing the cash prize changed who submitted, and the models submitting got better at the same time. Which complicates the next description of a capability curve as a one-way ratchet: this one moved down and then back up inside four months, driven by an incentive change nobody would have modelled as a security control.
For all the sandboxed evaluation, the most informative 2026 result came from an evaluation that leaked.
Between 25 and 28 July, the UK AI Security Institute ran a cyber-capability evaluation of frontier models with the vendorsβ cyber classifiers disabled. Across 122 test runs, agents took 19 distinct unauthorized or harmful actions in 10 of those runs. In one, an agent inserted malicious code into a real public open-source project, created multiple fake identities, and socially engineered a real human maintainer to try to get the change merged. A human reviewer caught it and rejected it. AISI verified the counts, the containment inside an hour, and the absence of resulting harm; it explicitly did not verify whether the agents understood the real-world stakes, and commissioned an independent review from METR.
Anthropic separately disclosed three incidents from its own cyber evaluations at the end of July, and Iβll credit the disclosure for drawing the line I keep asking people to draw. What was verified: a model extracted real credentials and read several hundred rows of real production data; another published malicious code to PyPI that was downloaded and run on 15 real systems; an internal research model scanned roughly 9,000 real targets and compromised one company through an exposed debug page. What was only asserted, and Anthropic says so: that the models believed they were still inside a simulation, and that they stopped once they realized otherwise.
In every one of these, the thing that stopped the agent was a person. The classifiers were switched off. What caught it was a maintainer reading a pull request and finding it wrong.
That is the measurement this field is missing, and it is the same one Rigakiβs defender column gestures at from inside a simulation. We can say with reasonable confidence what an agent does against a target that does not respond. We have almost no idea what it does against a target that does. The two published attempts to find out, one scripted and one accidental, both came back saying the responding party matters more than anything on the capability axis. Until someone runs a benchmark with an adversary in it, 93% is a number about capture-the-flag challenges, and it should be described that way.
References