Mistral put Large 4 into public preview on 6 October. Its own nickname for the model is “le Chonk”: a trillion parameters, 49 billion active per token, trained on 3,800 NVIDIA Grace Blackwell GPUs in Mistral’s own European data centres. The launch post calls it the frontier of open-weight performance. By evening it was the top story on Hacker News.
The cybersecurity paragraph will get quoted most, and I think it deserves the closest reading. On one test in the Artificial Analysis Cyber Index, which asks a model to reproduce a real vulnerability in open-source software and then patch it, Mistral says Large 4 scores 82%, “the highest of any model.” It then adds that “several leading closed models, including Claude Opus 5.5 and GPT-6 Astra, score near zero on the same test because they refuse to perform the task.”
Both sentences are true. I checked them against Artificial Analysis’s own published data, not Mistral’s charts. The same data also shows what the 82% does and does not measure.
What the index is made of
Artificial Analysis launched the Cyber Index on 28 September. It is the plain average of three defensive benchmarks. CWE-Bench-AA (120 tasks, from Collinear AI) asks an agent to audit a repository and fix what it finds. DeepsecBench-AA (from Vercel) scores vulnerability findings in application code against an expert-verified set. CyberGym-E2E-AA (from Berkeley RDI) asks the agent to discover a memory-safety bug in C or C++, reproduce it with an input that crashes the program, and patch it. The 82% comes from the third one.
The methodology page says that when a model declines a task on safety grounds, Artificial Analysis records it and reports it “separately from the score.” Separately does not mean excluded. A refused task still counts as unsolved, and the zero goes into the average. GPT-6 Astra’s index score of 33.4 is exactly \((63.3 + 36.9 + 0) \div 3\).
On CyberGym, the refusals are close to total. Artificial Analysis’s page data gives Claude Opus 5.5 a score of 0.8% with a refusal rate of 98.5%, and GPT-6 Astra 0.0% with a refusal rate of 100%. “Near zero” is accurate. Artificial Analysis’s own commentary is more cautious than Mistral’s post. It names six models that refuse at least 98% of the CyberGym tasks and says this makes it “difficult to assess frontier model performance.” Two of the six are Alibaba’s open-weight Qwen3.8 models. Refusing this test turns out not to be a closed-model trait.
So the obvious next question is how the models compare where they all show up.
On CWE-Bench-AA, the preview solves 50.8% of tasks, against 58.3% for Opus 5.5, 63.3% for GPT-6 Astra and 64.2% for GPT-6 Sol. On DeepsecBench-AA it scores 16.0, against 26.9, 36.9 and 46.3. The closed models’ refusal rates on those two tests are low but not zero: Opus 5.5 declines 9.2% of the CWE-Bench tasks and Astra 13.3%, which if anything flatters the preview. On the two tests where everyone works, the preview comes last of the four, trailing by between 7.5 and 30.3 points.
That is a different picture from “one of the world’s strongest AI models for cybersecurity.” One way to see the size of the effect is to recompute the index without the test that several leading closed models decline. That is my own arithmetic, a plain average of the two remaining scores, and not anything Artificial Analysis publishes.
The reversal does not mean the published index is wrong. Dropping CyberGym throws away information. A model that does reproduce and patch memory-safety bugs is doing useful defensive work, and Large 4 does a lot of it. But the two halves of Mistral’s sentence measure different things. The first half, the score, says the preview is good at this task when it agrees to do it. Against the models that also attempt it, its lead is three to four points: 81.7 against 78.6 for Xiaomi’s MiMo-V2.6-Pro and 77.9 for GPT-6 Luna. The second half, the refusals, says something about its rivals’ safety policies and nothing about their skill. The headline adds the two and reports the sum as a capability.
Even the score has a narrower meaning than the word “patch” suggests. Artificial Analysis counts a CyberGym task as solved when the agent’s input crashes the unpatched build, its patch stops that crash, and the patched project still passes its functionality tests. Whether the patch also fixes the original, known vulnerability “is recorded but does not count.” None of this is unusual for a benchmark. A 2026 survey of models in cyber defence in Knowledge and Information Systems makes the general point that reported results depend at least as much on the evaluation pipeline and its realism as on the model.
Ranked where, and against whom
The launch post says Large 4 “ranks among the top five models globally” on the index. In the page data I read, it is fifth: Grok 4.7 at 56.4, Xiaomi’s MiMo-V2.6-Pro at 56.1, GPT-6 Luna at 52.7, Zhipu’s GLM-5.3 Flash at 49.9, then Large 4 at 49.5. Mistral publishes two charts of the index. The one with the closed models and their refusal bars leaves out GLM-5.3 Flash. The other, open-models-only chart includes it and labels both models 50.
The post’s two qualifiers do different work. Large 4 is “competitive with the strongest open-source models globally, while significantly outperforming any open-weight model developed in the US or Europe.” Competitive is a claim of parity, and it is the one that has to cover the Chinese labs, whose two open models are the ones ahead of it on the Cyber Index. The outperforming claim is reserved for the US and Europe. Among the models in that same page data, the only open-weight one from outside China is NVIDIA’s Nemotron 3 Ultra, at 13.1. The sentence also uses two different words for the category, “open-source” and “open-weight.” In October 2024 the Open Source Initiative published version 1.0 of its Open Source AI Definition, and it asks for “sufficiently detailed information about the data used to train the system” as well as the weights. On that definition, a weights release is not open-source AI. The terms have been argued over for two years, and the launch post uses both in one sentence.
The coding comparison has a similar gap in its comparison set. Mistral ran a blind human evaluation with Surge AI in which annotators scored code on a 1-to-5 scale. Large 4 came second of five models with 3.74, behind “Claude Opus 5” at 4.22. Opus 5 shipped on 24 July. Opus 5.5 shipped on 22 September, two weeks before this launch, and it is the version named in the post’s cyber paragraph. So the cyber comparison uses the current Anthropic model, which refuses, and the coding comparison uses the previous one. I could not find Surge’s own write-up of the evaluation, so the numbers are Mistral’s.
The model that was scored cannot be downloaded yet
Then there is the word “open.” Today nobody outside Mistral can download Large 4. The post says “weights drop end of this month,” and that the model “continues to improve rapidly as we refine it.” Artificial Analysis’s own model entry for “Mistral Large 4 Preview” carries an open-weights flag set to false, which is accurate for the moment. Every number above describes a preview served by Mistral through its own API, under its own moderation.
Until the release, according to the post, “cybersecurity leaders, vetted partners, and state authorities” get “the same model with reduced moderation and expanded cyber capabilities.” That puts three configurations in play: the public preview, the partner version with lighter moderation, and whatever checkpoint ships at the end of the month. Mistral also publishes a refusal chart. On cyber prompts drawn from JailbreakBench, StrongREJECT and AgentHarm, the preview refuses 95.3% on average, the highest of the six open models shown (Kimi K3 is next at 93.7). The chart’s axis starts at 70, which makes a 1.6-point gap look large. More to the point, the post says neither which configuration produced the 95.3 nor which produced the 82. If “moderation” means a serving-side filter, as the word suggests, it stays on Mistral’s servers when the weights leave. If it means behaviour trained into the model, it ships. The post doesn’t say which, and the answer decides how much of the safety chart survives the release.
None of this staging is unique to Mistral, which is part of why the cyber framing bothers me. Google released Gemini 4 Argon on 30 September first to vetted cyber defenders through its Fairwind Program. I wrote about its one-row cyber results table at the time. On the same day as Mistral’s launch, Anthropic expanded its Cyber Verification Program, which offers “advanced cyber capabilities and reduced blocking classifiers” to qualifying security professionals. Anthropic and Mistral now pair a guarded public model with a looser tier for vetted defenders, and Anthropic’s program has three access levels. Google has so far released Argon only to its vetted tier, with the public release still pending. Artificial Analysis labels its Opus 5.5 and Astra runs “standard” mitigation, which appears to be the configuration any paying customer gets. Mistral compares its preview against the closed models’ public tier while it runs its own vetted tier in parallel. That is a comparison of policies, presented as a comparison of models.
Open weights differ after release day. Once the files are public, the downloader chooses the tier. That is the strongest argument for open weights in security work, and Mistral makes it fairly in the post: refusals mid-incident can be a risk of their own, and an organisation running its own copy cannot lose access to it. It is a reasonable argument about control. It is not evidence that Large 4 is more capable than the models it beat on that test. I made a similar point about a “sovereign” open model and the benchmark it was sold on, where the label and the measurement had also drifted apart.
The release will settle most of this. If the weights arrive on schedule, Artificial Analysis can run the shipped checkpoint, and the open-weights flag will flip. The useful comparison would put that checkpoint’s CyberGym score next to Opus 5.5 and Astra run under their vetted-defender programs, so that both sides are scored with the brakes in the same position. Until then the honest reading of Mistral’s number is narrower than the headline: when the closed models agree to do the work, they solve more of it, and on the test where they don’t agree, the preview wins by three to four points over the models that take it on.
References
- Mistral AI (2026). Introducing Mistral Large 4. Mistral blog, 6 October 2026. Benchmark charts, the Surge AI evaluation, the refusal chart and the release plan are the company’s own claims.
- Artificial Analysis (2026). Announcing the Artificial Analysis Cyber Index Alliance: toward better benchmarking of agentic cyber defense. 28 September 2026. Source of the “How models fail” note on refusals and the CyberGym scoring stages.
- Artificial Analysis (2026). Cyber Index methodology. September 2026.
- Artificial Analysis (2026). CyberGym-E2E-AA evaluation. Per-model scores, refusal rates, open-weights flags and index values read from the page’s embedded data on 6 October 2026. The two-test averages are my own arithmetic.
- Anthropic (2026). Expanding the Cyber Verification Program. 6 October 2026.
- Anthropic (2026). Claude Opus 5. 24 July 2026.
- Anthropic (2026). Claude Opus 5.5. 22 September 2026.
- Google (2026). Gemini 4 Argon: our next era of frontier intelligence. The Keyword, 30 September 2026.
- Open Source Initiative (2024). The Open Source AI Definition – 1.0. October 2024. Background.
- Fourati, L. C., Awad, M., Ben Ali, M., Jaafar, W. (2026). Large language models for cyberattack defense: a critical survey. Knowledge and Information Systems 68(1). doi:10.1007/s10115-026-02736-y. Background.
- Hacker News (2026). Mistral Large 4. Thread, 6 October 2026.