Gemini 4 Carbon Claims Three Security Records. One of Them Is on a Public Leaderboard.

On 28 October Google announced Gemini 4 Carbon, the successor to Gemini 4 Argon, in a post by Koray Kavukcuoglu. Carbon goes first to “a subset of cyber defenders” through the Fairwind Program, with wider release promised but not dated. The post leads on security. “Carbon achieves a state-of-the-art 88.2% on SEC-bench Pro,” it says, and it “sets a new state-of-the-art on the popular exploitation benchmark ExploitGym (54.8%).” For patching, “Carbon reaches 71.0% on Collinear’s CWE-bench.”

When Argon launched I wrote that its results table had one cyber row. Carbon’s post has three cyber numbers, which is more. So I went to check each one where the benchmark keeps its own results.

Three numbers, three kinds of evidence

SEC-bench Pro is a long-horizon security benchmark from a paper posted to arXiv in May. Its public leaderboard has an overall board of 344 tasks across V8, Firefox and Linux. The top entry there is GPT-5.5 at 58.4%, solving 201 of the 344. No Gemini model appears on any of its boards, so Carbon’s 88.2% is Google’s own run, reported in a blog post.

ExploitGym also comes from a May arXiv paper, and its repository describes 869 task instances. The repository page has no leaderboard and no scores, so the 54.8% has nothing public to sit beside.

CWE-bench is different. It is run by Collinear on a private set of 120 tasks, and its v1 leaderboard lists Gemini 4 Carbon first, at 71% programmatic pass@1. That is the one claim in the post that a reader can check against a table the benchmark’s owner publishes.

Gemini 4 Carbon's three cyber claims and where each can be checked A ledger with three rows. SEC-bench Pro: Carbon claims 88.2 percent; the public leaderboard does not list Carbon or any Gemini model, and its top entry is GPT-5.5 at 58.4 percent of 344 tasks. ExploitGym: Carbon claims 54.8 percent; the repository has 869 tasks and publishes no leaderboard. CWE-bench: Carbon claims 71.0 percent; the public v1 leaderboard lists Carbon first at 71 percent on 120 tasks. THREE CLAIMS · WHAT EACH BENCHMARK PUBLISHES · CHECKED 29 OCT 2026 Benchmark Claimed On the benchmark's own public board SEC-bench Pro ExploitGym CWE-bench v1 88.2% 54.8% 71.0% Carbon not listed; no Gemini model on any board. Top entry: GPT-5.5, 58.4% (201 of 344 tasks) No public leaderboard or scores. 869 task instances in the repository Listed first, 71% programmatic pass@1. 120 held-out tasks, four trials per task Claims from Google's 28 Oct post. Boards and repositories read on 29 Oct.
Two of the three records are Google's own measurements with no public table to compare against. The third is on a leaderboard the benchmark's owner runs.

The checkable one is a three-point lead

On CWE-bench v1 the ordering is tight. Carbon has 71%. Three models tie at 68%: Grok 4.7, GPT-6 Astra and Google’s own Gemini 4 Argon. Claude Opus 5.5 has 67%. The Argon report, which describes the benchmark in detail, says each model runs four independent trials per task and that Pass@1 is the mean success rate across all 120 tasks.

On 120 tasks, one task is worth about 0.83 points, so Carbon’s lead over the next three models is about three and a half tasks on average. Four trials per task steady the run-to-run noise. They do nothing about the fact that 120 tasks is a small sample of what security audits look like; by my arithmetic the binomial standard error of a 70% score on 120 tasks is a little over 4 points, larger than the lead. The benchmark’s own judge panel, scored separately, puts Carbon at 71% and GPT-6 Astra at 66%, which is a wider gap, and the ordering below the top changes between the two columns.

CWE-bench v1 leaderboard, programmatic pass@1, top six models Horizontal bars on a 0 to 100 percent scale. Gemini 4 Carbon 71 percent, Grok 4.7 68, Gemini 4 Argon 68, GPT-6 Astra 68, Claude Opus 5.5 67, Claude Fable 5.1 58. One task of 120 is worth about 0.83 points. CWE-BENCH V1 · PROGRAMMATIC PASS@1 · 120 TASKS · COLLINEAR LEADERBOARD Gemini 4 Carbon Grok 4.7 Gemini 4 Argon GPT-6 Astra Claude Opus 5.5 Claude Fable 5.1 71% — CWE-bench v1 leaderboard, read 29 Oct 2026 68% — CWE-bench v1 leaderboard, read 29 Oct 2026 68% — CWE-bench v1 leaderboard, read 29 Oct 2026 68% — CWE-bench v1 leaderboard, read 29 Oct 2026 67% — CWE-bench v1 leaderboard, read 29 Oct 2026 58% — CWE-bench v1 leaderboard, read 29 Oct 2026 71% 68% 68% 68% 67% 58% One task of 120 is about 0.83 points. Carbon's lead over the 68% tier is about 3.5 tasks.
Carbon is first on the one board that lists it, by a margin smaller than the sampling error of a 120-task set.

The other two have no table to sit beside

The SEC-bench Pro comparison is the hardest to read. Carbon’s 88.2% is 30 points above the best entry on the public board, which is GPT-5.5 at 58.4%. That gap is not a like-for-like comparison. The public board’s newest entries are older models, and none of the October 2026 frontier models is on it, so the 58.4% says more about who has submitted than about where the field is. The comparison a reader would want, Carbon next to GPT-6 Astra and Claude Opus 5.5 on the same 344 tasks with the same harness and budget, is not published anywhere I could find.

SEC-bench Pro: public leaderboard entries and Carbon's self-reported score Horizontal bars on a 0 to 100 percent scale. Public overall board: GPT-5.5 58.4 percent, GPT-5.4 39.0 percent, Claude Opus 4.6 30.8 percent. Gemini 4 Carbon's 88.2 percent is drawn as an outline because it is self-reported in Google's post and does not appear on the board. SEC-BENCH PRO · OVERALL BOARD, 344 TASKS · VERSUS CARBON'S CLAIM Gemini 4 Carbon (claimed) GPT-5.5 GPT-5.4 Claude Opus 4.6 88.2% — Google, 28 Oct 2026, not on the public board 58.4% (201/344) — SEC-bench Pro board, read 29 Oct 2026 39.0% (134/344) — SEC-bench Pro board, read 29 Oct 2026 30.8% (106/344) — SEC-bench Pro board, read 29 Oct 2026 88.2% 58.4% 39.0% 30.8% Outlined bar: self-reported, not on the board. Filled bars: the board's top three entries. The board lists no October 2026 frontier model, so the gap is not like for like.
The 30-point gap is between Google's run and a board whose newest entries are older models. It shows what has been submitted, not where the field is.

ExploitGym has the same gap with less around it. The repository describes the task set and points to a submission format, but no scores are posted, so “a new state-of-the-art” is a claim about runs that only Google has published a number for.

The post does link a “Safety and Security report.” The link goes to the Gemini 4 Argon Safety & Security Report, dated October 2026 and 217 pages long. A text search of it finds no mention of Carbon, and none of SEC-bench Pro or ExploitGym. Its CWE-bench section is about Argon’s 68%. So for two of Carbon’s three security records, the post is the whole published method.

What would make the records checkable

None of this says Carbon is weak at security work. A 71% on a held-out audit-and-patch set is a strong result, and Google is putting the model in front of defenders first, which is where a security-capable model is most useful. But the post presents three records in one breath, and they rest on three very different kinds of evidence.

The fix is cheap and has been asked of the field for years. In 2023 a group of sixteen researchers, two of them at DeepMind, argued in Science that aggregate metrics and lack of access to results limit our understanding of what AI systems can do. For Carbon, publishing the results would mean submitting to the SEC-bench Pro board, posting ExploitGym results in the benchmark’s own format, and shipping a Carbon section in the safety report. Until then, the CWE-bench row is the one I would quote.

References

  1. Kavukcuoglu, K. (2026). Introducing Gemini 4 Carbon. Google, 28 October 2026.
  2. Google DeepMind (2026). Gemini 4 Argon Safety & Security Report. October 2026, 217 pages. Linked from reference 1 as its “Safety and Security report.”
  3. SEC-bench (2026). SEC-bench Pro leaderboard. Overall board, read 29 October 2026; paper: SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?, arXiv:2605.26548, 26 May 2026.
  4. ExploitGym (2026). sunblaze-ucb/exploitgym. Repository, read 29 October 2026; paper: ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?, arXiv:2605.11086, 11 May 2026.
  5. Collinear (2026). CWE-bench v1 leaderboard. Read 29 October 2026.
  6. Burnell, R., Schellaert, W., Burden, J., Ullman, T. D. et al. (2023). Rethink reporting of evaluation results in AI. Science 380(6641), 136–138, 14 April 2023.
  7. Gautam Parab (2026). Gemini 4 Argon Is Held Back Over Cyber, and Its Results Table Has One Cyber Row. 30 September 2026.