On 15 September Google released two voice models under one name. Gemini 3.8 Live is the fast, cheap one. Gemini 3.8 Live Extended Thinking reasons while it talks. The announcement credits the second with first place on Artificial Analysis’ Speech to Speech Index at 82.6, 68.6% on τ-Voice and 35.1% on what it calls “Sierra’s τ-Voice-banking benchmark.”
I went through the evaluation document Google links from its model card, and then through the leaderboards that document cites. Three things came out of that reading. The headline numbers belong to one of the two models. Its lead over OpenAI is smaller than the noise in the measurement. And the benchmark family behind the agent scores was designed around a reliability metric that nobody in this release reports.
Google’s methodology PDF charts both models side by side. On τ-Voice, the Extended Thinking model scores 68.6%. The base Gemini 3.8 Live scores 30.1%, lower than the 37.7% Google’s own previous-generation Gemini 3.1 Flash Live reached at its high setting. The same PDF says the τ-Voice runs used “thinking high” for both model IDs, so on Google’s account the gap isn’t down to one of them being run at a lower setting.
The distinction matters because of where each model ships. The base model is the one rolling out “for everyone” in Search Live, and it is priced for that job: Artificial Analysis lists it at $0.84 per hour of input audio, against $3.50 for Extended Thinking and $5.83 for OpenAI’s GPT-Live-1 Astra.
The base model does win something. Google says it took second place in the Speech Agent Arena, where people compare two hidden models in live conversations, and the leaderboard confirms it: Elo 1083, just behind Gemini 3.1 Flash Live (Minimal) at 1096. The arena prints its own 95% intervals, ±30 and ±27, and gives the base model a rank range of first to fourth. Its task success rate there is 93.2%. That sounds like a different model from the one that completes 30.1% of τ-Voice tasks, until you look at the arena’s scenarios: booking a dental appointment and ordering takeout. τ-Voice, as Sierra designed it, asks an agent to do things like change a flight under airline policy, while a simulated caller who can interrupt speaks over audio mixed with noise and telephone compression. Both numbers can be true at once. The Extended Thinking model, for what it’s worth, sits tenth in the arena at 990.
Here are the three tables Google cites, drawn on one scale.
The τ-Voice gap is 0.7 points. Artificial Analysis, which runs the test, is open about how it gets its numbers, and its methodology page says each score is the mean of three independent trials over 278 scenarios: 50 airline, 114 retail and 114 telecom. A note on the leaderboard adds that GPT-Live-1 Astra, the runner-up, was scored on one trial. So the headline lead compares a three-run average against a single run. If the tasks are treated as independent, one pass over 278 of them at about 68% has a standard error near 2.8 points (my arithmetic). The lead is a quarter of that. On these numbers the two models are tied.
The Speech to Speech Index lead is 1.1 points, and the index needs care anyway. Google’s post calls it a “Quality Index”; Artificial Analysis, which built it, calls it the Speech to Speech Index. It has also changed shape twice since it launched in June. Version 2.0, dated August, weights four components equally: Big Bench Audio, τ-Voice, arena preference and arena task success. The Extended Thinking model ranks tenth on arena preference, so that component isn’t what puts it first.
The banking number is less settled still. Google’s blog calls the benchmark τ-Voice-banking. Its own PDF titles the chart “τ³-Banking Leaderboard” and describes a test in which agents search a large unstructured knowledge base and chain tool calls through banking workflows. It is the banking domain Sierra introduced as τ-knowledge in 2026: 698 documents, about 195,000 tokens, and tasks that need 9.5 tool calls on average. Artificial Analysis’ version runs 97 of those tasks and reports pass@1 “averaged across repeats,” without giving the number of repeats. The Gemini row doesn’t appear in the leaderboard text I pulled, so for Google’s 35.1% against Astra’s 32.0% I have only Google’s chart. A single pass over 97 tasks carries a standard error of nearly five points.
Google’s evaluation document settles one question in its first line of methodology: “All Gemini scores are pass @1 except where otherwise noted.” No exception is noted for the agent benchmarks. Pass@1 is the average chance that one attempt succeeds. It is what nearly every leaderboard reports, and for a voice agent it measures the wrong thing, which the benchmark’s own designers pointed out.
When Sierra introduced τ-bench in June 2024 (Yao et al.), the paper proposed a companion metric, passk: the chance that an agent succeeds on all k independent tries at the same task, averaged over tasks. The reasoning was that a customer-service agent that solves a problem sometimes is not an agent you can deploy. In that paper GPT-4o completed about 61% of retail tasks on one try, and its pass8 fell to about 25%. τ-Voice inherits the same tasks, tools and evaluator. Its launch post from 1 May says they are byte-for-byte identical to the text version’s.
Passk can’t be recovered from pass@1 alone, but it can be bounded. If every task is equally hard and tries are independent, passk is the single-attempt rate raised to the kth power. If every task is all-or-nothing, where the agent always solves it or never does, passk equals pass@1. Real benchmarks fall between those bounds. Artificial Analysis already ran three τ-Voice trials of the Extended Thinking model, so the data for pass3 exists. It isn’t published, and on 68.6% the bounds allow anything from 32.3% to 68.6%.
The published pairs show where measured agents land.
IBM Research published the cleanest recent example on the same day as Google’s launch. A ReAct agent on GPT-4.1, run five times at temperature zero over 168 AppWorld tasks, succeeded on 77.4% of runs and on all five runs for only 53.0% of tasks. The underlying paper calls that 24-point shortfall the consistency gap. IBM traces it to the serving side more than to sampling: on a hosted endpoint the probabilities drift slightly between calls, so a decision that was close to a tie can go the other way on the next run, and a multi-step task gives it many chances to. IBM’s guidelines narrowed the gap to 12 points, 81.0% against 69.0%.
Sierra’s own τ-knowledge post gives the banking-specific version. When the domain launched in March, GPT-5.2 at high reasoning passed 25.5% of tasks on the first try and 9.3% on all four. By May, GPT-5.5 at its top reasoning setting reached 37.4% and 20.6%. Those models, harnesses and task sets differ from Google’s, so none of this is an estimate for Gemini. As an illustration only: if Gemini’s 35.1% kept the same share GPT-5.5 kept, its pass4 would be about 19%, roughly one banking task in five handled correctly four times running.
In September 1964 the Bell System Technical Journal gave an entire issue to the No. 1 Electronic Switching System, the first large stored-program telephone exchange. Two of its papers deal with dependability, and they invert the usual computer priorities. In a computing centre, the overview paper argues, a stopped machine is a nuisance, because the job can be rerun, and a wrong answer is the disaster. In a telephone office it runs the other way: a total failure is the disaster, and a mishandled call is a nuisance to the customer, who has to redial. The maintenance-plan paper sets the objective that followed: no more than two hours of total downtime in the office’s 40-year design life.
A mishandled call could be written off as a nuisance for two reasons. The customer usually knew the call had failed, and the redial was close to an independent second draw from a very reliable machine. Retrying until something works is the world pass@k describes. The τ-bench paper says as much: pass@k suits tasks where a result can be checked, and customer service is not one of them.
A voice agent breaks the first condition more often than the second. τ-Voice scores the final state of the database, not the conversation, and Sierra’s launch post names the risk it was built to catch: agents that hold a pleasant conversation while quietly failing the task. A caller whose refund was filed against the wrong order, or not filed at all, has no reason to call back until the statement arrives. Nothing prompts a redial, so what the caller experiences is the first attempt and every attempt after it, which is passk.
That’s why the missing number matters more for this release than the 0.7 points. Google’s post says the models give developers and enterprises “the building blocks for reliable, production-ready voice agents.” Reliability has a measure in the benchmark family Google chose to cite, and Artificial Analysis appears to have enough τ-Voice trials in hand to compute part of it. When I wrote about a chess honeypot yesterday, the argument turned on denominators nobody had published. This is the same problem pointed at deployment. A 68.6% can mean an agent that solves two-thirds of problems every time, or one that solves most problems about two-thirds of the time. Those are very different products, and the published number cannot tell them apart. The harness question from Astra’s two ARC-AGI-3 scores applies here too: Google’s PDF says ServiceNow ran its EVA-Bench numbers through Google’s enterprise agent platform, while the τ-Voice and banking runs went through the public API.
None of this needs new experiments. Artificial Analysis could publish pass3 from the τ-Voice trials it has already run and score the runner-up on three trials like everyone else. Google could put the model’s full name next to every number.
References
Standard errors, the passk bounds and the 19% illustration are my own calculations from the published figures.