Google launched the Gemini Decisions API on 11 November, and with it its first decision model, Gemini Decision 1. The launch post gives the contract and a price. You send one input of up to 256K tokens (text, images, audio, video or PDF) with up to 1,000 questions, each a yes/no, a choice among 2 to 255 options, or a rubric score. The model “scores your candidate options in a single forward pass, returning probabilities and confidence scores in 150-250 ms end-to-end latency at $0.10 per 1M input tokens with zero output tokens billed.”
Beyond that, the post makes two claims. Gemini Decision 1 “leads across the suite, with the lowest calibration error,” and it is “nearly 5x cheaper across the shared benchmark suite than Gemini 3.5 Flash-Lite with the Interactions API.” I’ve written about this category three times, about Jev’s speed headline, about two decision models claiming first place on one benchmark, and about what a probability over five tokens means. This time I read Google’s benchmark table row by row and set it next to the one outside evaluation published the same day.
The table Google published
The comparison is an image in the launch post, so the numbers can’t be copied out of it, but they can be read. It has four columns: Gemini Decision 1, TypeSafe Jev 1.14, GPT-6 Luna through OpenAI’s Decisions API, and Gemini 3.5 Flash-Lite through Google’s Interactions API with minimal thinking. There are nine accuracy rows, one calibration row and one cost row. A footnote says refusals, safety blocks and requests a model couldn’t accept count as wrong, and every score uses the full benchmark as the denominator. That’s the strict way to score, and I’m glad it’s stated.
Jev and Luna have no entries for the two audio and video rows, MMAU and Video-MME, so the calibration and cost rows both say “7 shared evals.” On those seven, Decision 1 beats Jev on five. Jev wins Mind2Web Multimodal, 86.8% to 86.2%, and When2Call, 82.0% to 79.9%, and Google’s table bolds Jev’s cell in both rows. When2Call tests whether a model should call a tool, ask a follow-up question or admit it can’t answer (Ross et al., 2025). Tool selection is the first use the launch post lists.
The JevBench row carries one more qualifier: it’s labeled “Public split.” When I looked at JevBench in October, its release had 1,500 decisions, 300 public and 1,200 sealed, and the sealed set is the one its official boards rank. So 92.2% against 90.0% is a result on the 300 items anyone can download. That’s a fair thing to report. It just isn’t a JevBench leaderboard position.
What “nearly 5x cheaper” is measured against
The cost row is “Total price” across the seven shared evaluations. Jev costs $0.68, Gemini Decision 1 $0.88, Luna $0.93 and Flash-Lite $4.14. Divide $4.14 by $0.88 and you get 4.7, which is the “nearly 5x.” It’s arithmetic on the right row, but the denominator is Google’s own general-purpose model with generated output, not either of the other decision models in the same table. Against the cheapest column, Decision 1 costs about 29% more.
For someone deciding whether to replace an LLM call with a decision call, the Flash-Lite comparison is still the useful one. That’s the swap the post is selling, and a near-5x saving with higher accuracy on most rows is a good result. For someone choosing among decision models, the useful comparison is the cheapest column, and it isn’t Google’s.
The outside run the same day
Artificial Analysis posted its own evaluation the same day. On its Decision Index v2.1, Gemini Decision 1 scores 66, “in line with Elephant 9B (@baselabs) and Jev 1.14 (@typesafeai), and 7 points behind leaders Drex Max v1.1 (@NaceAI) and GLiDE v1.2 (@fastinoAI), which both score 73.” In its chart that ties for seventh among 18 systems. The index is a different suite, weighted across AA-Paperclip, EvalSafe-O*NET-AA, When2Call, AA-Omniscience-Decision, HLE-MultiChoice and AA-Symmetry, and it isn’t trying to replicate Google’s. It’s the only outside number available on day one.
The two sources agree on direction where they overlap. Decision 1 leads on knowledge and reasoning, 66 on AA-Omniscience-Decision and 49 on HLE-MultiChoice. It trails on Artificial Analysis’s new extraction benchmark, where “it answers all 15 questions correctly on 33% of documents vs. 63% for Drex Max v1.1.” On cost the gap is wider than in Google’s table: $345 per million tasks, against $156 for Jev and $103 for Drex Max. Artificial Analysis puts that down to price per token, $0.10 per million input tokens against $0.03 for Drex Max, and notes that Luna lands close, at $339.
The latency claim checks out. Google said 150 to 250 ms end to end, and Artificial Analysis measured 211 ms per task across its index. That’s slower than Jev at 92 ms and faster than Drex Max at 283 ms.
Two calibration numbers that seem to disagree
The calibration row is where Google’s table makes its strongest claim. Top-1 expected calibration error across the seven shared evaluations is 0.0250 for Decision 1, 0.0494 for Jev, 0.0721 for Luna and 0.2104 for Flash-Lite. That’s half Jev’s error. Artificial Analysis reports a Brier score instead, “the average squared gap between the probability a model assigns to its chosen answer and whether that answer was correct,” and gets 0.18 for Decision 1, 0.16 for Drex Max and 0.19 for Jev. On that number Decision 1 and Jev are almost level.
The two numbers can both be right, because they measure different things. Expected calibration error bins predictions by confidence and asks whether a model’s 80% answers come out right about 80% of the time. A model that is never sure of anything and is right as often as it says scores well. The Brier score adds a second ingredient. Its decomposition into calibration and refinement components goes back decades in the forecasting literature (Blattenberger and Lad, 1985), and refinement rewards a model for pushing its probabilities toward 0 and 1 when it can. Halve the calibration term and the Brier score can still barely move if accuracy and sharpness are about the same. That fits these two models, which differ by one to two points of accuracy on most of Google’s rows. The survey by Silva Filho and colleagues (2023) covers calibration metrics and proper scoring rules as separate families of evaluation for this reason.
Which one matters depends on what you do with the score. If your code routes anything under 0.7 confidence to a human, calibration error is the number that tells you whether that threshold means what you think. If you want one number for how good the probabilities are overall, the Brier score is the better summary. Artificial Analysis also gives a per-task split that matters for routing. Decision 1’s Brier score on When2Call is 0.14, against 0.04 for Drex Max. That’s the tool-calling row again, the one where Jev also beat it in Google’s own table.
What I’d take from it
Google has shipped a decision model that’s competitive on accuracy, fast in the way it says, and cheaper than its own generative fallback by about the factor it claims. Those are real results, and the launch table is more honest than most. It bolds two rival wins and footnotes a strict scoring rule.
What it doesn’t do is put Decision 1 at the front of its own category on price. Google’s cost row has Jev cheaper, and the outside run puts Jev’s cost per task at under half Decision 1’s. The headline multiple is measured against another Gemini model. The routing benchmark is where the model looks weakest in both sources, and routing is the first use the launch post lists. If I were choosing a decision model for tool selection this week, I’d run When2Call-style items from my own traffic through Decision 1, Jev and Drex Max before reading any more launch tables, including this one.
References
- Google (2026). Gemini Decisions API: Fast, calibrated multimodal decisions. The Keyword, 11 November 2026, including the benchmark comparison table image.
- Artificial Analysis (2026). Gemini Decision 1 on the Artificial Analysis Decision Index, with the time-per-task follow-up. Posts and charts, 11 November 2026.
- Latent Space (2026). [AINews] D-Day (Decision Models Day). 11 November 2026.
- Blattenberger, G. and Lad, F. (1985). Separating the Brier Score into Calibration and Refinement Components: A Graphical Exposition. The American Statistician, 39(1), 26-32.
- Silva Filho, T., Song, H., Perello-Nieto, M., Santos-Rodriguez, R., Kull, M. and Flach, P. (2023). Classifier calibration: a survey on how to assess and improve predicted class probabilities. Machine Learning, 112(9), 3211-3260.
- Ross, H., Mahabaleshwarkar, A. S. and Suhara, Y. (2025). When2Call: When (not) to Call Tools. arXiv:2504.18851, NAACL 2025.