Within three days, two companies announced a “decision model” and pointed at the same benchmark. On 7 October, H2O.ai’s model card for H2O-Lightning-4B said the 4B-parameter model is “#1 on the official Composite Score” of open-weight systems on JevBench v1.6.1. On 9 October, Microsoft announced Microsoft-Decision-1, calling it the most accurate model in a 36-benchmark comparison and the fastest measured, “2.5 times quicker than H2O-Lightning-4B v1.1, the runner-up.”
Both announcements lean on a benchmark most readers have never seen. I wrote about the model that gave it its name a few weeks ago: Jev’s “193.6 times faster” turned out to depend on which rival sat in the denominator. This time the open question is who runs the leaderboard, and what “#1” is a rank of.
What a decision model is scored on
Microsoft’s definition is compact: unlike a language model that writes text, a decision model is “purpose-built to deliver structured outputs that software can immediately act on.” You hand it a record and a fixed set of options, and it returns a probability for each. H2O’s card describes the same contract (choice, yes/no, and ordinal score) and says each decision is one forward pass and one output token, so the cost is input tokens only.
That contract produces four things worth scoring, and JevBench scores all of them. Its methodology page calls them intelligence (chance-corrected decision accuracy, with tier weighting), calibration (whether the probabilities mean what they say), speed, and cost. Two summary scores sit on top. Capability is the mean of intelligence and calibration, ranked only for systems within the board’s cost and latency caps. Composite is a weighted harmonic mean of all four, with equal 25% weights, and squared penalties when intelligence falls below 50 or speed or cost below 50.
The harmonic mean matters. It punishes a weak axis harder than an average would, which is why a small, cheap, fast model can beat a larger and smarter one on the composite while losing on capability.
Who runs JevBench
The leaderboard is run by Benchmark Heaven, which describes itself as maintained “independently of TypeSafe AI,” the company that built Jev. The site’s footer calls it “a hobby project” and says payments go to the “one-person company behind this project.” The scorer and the public benchmark repository are on GitHub under the name fstandhartinger.
That is not a criticism. It does shape what “official” can mean. The submit page says anyone can submit a model with no account; the queue is free, and a paid fast lane promises results within 48 hours. The release has 1,500 decisions, of which 1,200 are sealed and 300 public. Sealed questions and predictions are not downloadable, and the methodology page says “complete independent reruns of the sealed evaluation require the official evaluation process.” So the board ran H2O’s model itself, on held-out items. Public-split experiments and the composite arithmetic can be reproduced, but nobody outside the board can rerun the sealed evaluation. The change log records that H2O-Lightning-4B v1.1 was added on 7 October as an add-request, one of two Qwen3.5-4B fine-tunes measured that day.
So H2O’s “#1” is a leaderboard entry, not a self-run. The scores come from the board’s own measurement. The ranking, though, is a choice, and the next section shows how much room that choice has.
One row, four ranks
In the board’s published data, H2O-Lightning-4B v1.1 scores 72.5 on the composite, ahead of the Jev reference row at 71.5 and Quyet-1.0-Large at 71.4. On capability it scores 75.0 and ranks fifth among open-weight systems. Quyet-1.0-Large leads that list at 81.7, and the board’s own page says so: “Quyet-1.0-Large leads the Jev-class systems with 81.7.” The methodology page says the open-weights board “leads with Capability,” and the board’s FAQ calls the composite “secondary.”
H2O’s card leads with the composite. Microsoft’s chart labels the same model “#5 on JevBench” and Quyet “#1 on JevBench,” which is the capability ordering. Both are true statements about one board. “Official #1” is also an open-weights claim: when the data file merges the open and hosted-API rows, the same composite score ranks second, behind a hosted API row, Levanto’s Sage 1.3.0, at 74.0. Neither company misreports a number. Each picked the ordering that favors it, and both orderings are defensible readings of the same table.
The 4B model’s strength is specific. Its calibration (90.0) matches the 31B Quyet’s, it is faster and cheaper than the larger models, and it gives up 13 points of intelligence to get there. The cost figure is also a labeled estimate: the board carried a cost measured for the earlier v1.0 across to v1.1, as it does for every Qwen3.5-4B row.
What Microsoft actually measured
Microsoft-Decision-1 is a post-trained Qwen3.5-9B, available in Foundry and on OpenRouter, and Microsoft says it will be rebased on other models, including MAI and OpenAI ones. The headline accuracy is 83.5%, a mean across 36 benchmarks and 147,137 questions that Microsoft chose, with some public and some private. The article says they were “kept blind from training,” which I can’t check, since the private ones aren’t published. In its chart, Jev sits at 82.3, Quyet at 81.9, and H2O at 77.2, seventh of the eight ranked rows. The margin over Jev is 1.2 points, and the article gives no interval. On calibration, Microsoft’s own chart puts its model third, 1.5 behind Jev.
An update note says the post was revised “to add benchmarks for Jev on accuracy and calibration.” The first version, in other words, left Jev’s accuracy and calibration off the chart.
I found no Microsoft-Decision-1 row on the JevBench boards. The published data file for v1.6.1 has no Microsoft decision model, as of 9 October. Its only Microsoft-origin rows are untuned base-model controls such as raw Phi-4 mini. Microsoft says it “took several of the top public models on the popular open leaderboard JevBench” and ran them on 36 further benchmarks. That is a re-evaluation of other people’s models on a set of 36 benchmarks Microsoft assembled. It is a fair design for checking generalization, but the result is not a JevBench rank, and the article does not claim one.
The speed claim
The speed claim is the cleanest to check, because the chart’s footnote names its source: “JevBench v1.6.1 adjusted median (p50).” The board’s adjustment for self-hosted models is the measured time times two plus 0.15 seconds, which the data file labels “assumed.” For H2O-Lightning-4B the measured median is 29 ms; the adjusted figure is 209 ms, and Microsoft charts 210. Microsoft-Decision-1 was timed at 85 ms through Foundry.
Against the 210 ms figure, the ratio is 2.5. Against the board’s measured 29 ms, it inverts: H2O’s model is about 2.9 times quicker. The comparison is not clean in either direction. The 29 ms comes from the board’s own raw field, on that system’s hardware. Microsoft’s 85 ms includes a service call. Both raw and adjusted fields are in the board’s published data, which is where I checked the arithmetic.
Nor does the 35-times-faster-than-GPT-6-Sol figure depend on any of this; that comparison is against a generating model. The ranking against other decision models is where the adjustment matters.
What I’d take from it
A decision model is judged on more axes than a language model, and every axis has a knob: the weights, the caps, the latency basis, the set of rows, the benchmarks chosen. Each launch set those knobs in its own favor. One detail from the methodology page shows how young the field is: it notes that “another project named JevBench tests metamorphic coherence,” and that it is unrelated to this board. The name is already in use for two different things.
My reading is that H2O’s claim is the more defensible of the two, because it points at a row the board itself measured and publishes the arithmetic. It is also a claim about the secondary score. Microsoft’s accuracy lead rests on 36 benchmarks it picked, a 1.2-point margin, and a latency ratio that flips depending on which of the board’s two numbers you use. None of this means the Microsoft model is slow or inaccurate; 85 ms is fast. H2O’s rank is a third-party measurement, though of the secondary score on a board with a single maintainer. Microsoft’s “fastest” and its accuracy lead are self-run, and Microsoft-Decision-1 does not appear on the board. A submission would settle part of it.
References
- Srivastava, A. (2026). Introducing Microsoft-Decision-1, our model for fast decision-making. Microsoft Command Line, 9 October 2026. Includes the embedded accuracy, latency and calibration chart and an editor’s note on a later update; read 9 October 2026.
- H2O.ai (2026). H2O-Lightning-4B. Hugging Face model card, “#1 on JevBench” section dated 7 October 2026; read 9 October 2026.
- Benchmark Heaven (2026). JevBench open-weights leaderboard. Release v1.6.1; read 9 October 2026.
- Benchmark Heaven (2026). Decision model benchmark methodology, JevBench v1.6.1. Reviewed 9 October 2026.
- Benchmark Heaven (2026). Submit a model for evaluation. Read 9 October 2026.
- Benchmark Heaven (2026). JevBench latest aggregate data. Revision v1.6.1, artifact SHA-256 5cd8c1332226ea9c179883d2285c0d8b8cc593ae38f2645197364a7ddd9c83c1; source for the merged-row ranks, raw and adjusted latency fields and cost bases; read 9 October 2026.
- Gautam Parab (2026). Jev Is 5 to 200 Times Faster, Depending on the Denominator. 20 September 2026. Background.