Ninety-Nine Percent of Five Tokens

Nish Tahir’s tutorial “Build your own decision model” went up on 10 October and collected 479 points on Hacker News (Tahir, 2026; Hacker News, 2026). It is short and clear. You take Qwen3-1.7B, a small open model, and give it a multiple-choice question with options A to E. You run one forward pass, keep the logits of the five letter tokens, throw away the rest of the vocabulary, and take a softmax over what is left. The largest of the five is the answer. Asked what colour the sky is, the model puts 0.9988 on B, Blue.

The commercial decision models sell the same contract. The two I compared on JevBench last week both take a fixed set of options and return a probability for each, and one of them describes each decision as a single forward pass and one output token. Tahir’s version lets you watch the mechanism, and he is careful with it. He writes that without extra training the scores probably reflect the model’s confidence about the next token rather than the probability that the answer is correct. Then he shows it.

The table

On a holdout sample of 1,221 CommonsenseQA questions the masked model got 725 right, an accuracy of 59.4%. A quick finetune raised that to 62.4%. Tahir then binned the base model’s answers by their top score. 809 of the 1,221 answers, about two thirds of the set, landed in the 0.9 to 1.0 bin with an average score of 0.9855, and they were right 70.1% of the time. The 121 answers in the 0.8 to 0.9 bin averaged 0.856 and were right 47.1% of the time. A question with no single right answer (where would you most likely find a bat: cave, baseball game, attic, zoo or sporting goods store?) gets 0.9978 on Cave.

A table like that is the oldest check in forecasting: when you say 90%, does it happen nine times in ten? Glenn Brier proposed a score for weather forecasts stated as probabilities in a three-page note in the Monthly Weather Review in January 1950, and checking the stated chances against what happened is part of what it measures (Brier, 1950). Run Tahir’s bins through the usual summary, the expected calibration error, which weights each bin by its count and averages the gap between score and accuracy, and I get 0.286. The mean top score across all 1,221 answers is 0.880, against an accuracy of 0.594. Both are my arithmetic from his table; he does not report them.

His fix is temperature scaling. You divide all five logits by one constant before the softmax, which flattens the distribution. He fits \(T \approx 3.80\) and the bins fall into line. The same arithmetic on his second table gives an expected calibration error of 0.041, and the top bin now holds 109 answers that average 0.933 and are right 95.4% of the time.

Masked softmax scores against accuracy, before and after temperature scaling Reliability plot of Qwen3-1.7B on 1,221 CommonsenseQA questions, from Nish Tahir's two bin tables. Before scaling, filled circles sit well below the score-equals-accuracy diagonal: bins averaging 0.38 to 0.86 are right 27 to 47 percent of the time, and the largest bin, 809 answers averaging 0.9855, is right 70.1 percent of the time. After temperature scaling with T of about 3.80, hollow circles sit close to the diagonal, with the top bin of 109 answers averaging 0.933 and right 95.4 percent of the time. Expected calibration error is 0.286 before and 0.041 after, the writer's arithmetic from the tables, both computed on the same 1,221 questions. TOP SCORE VS SHARE CORRECT · QWEN3-1.7B · 1,221 COMMONSENSEQA QUESTIONS · TAHIR, 10 OCT 2026 Share correct 0 0.5 1.0 0 0.5 1.0 Top score, bin average Before: 3 answers, score 0.283, right 0% Before: 26 answers, score 0.376, right 26.9% Before: 41 answers, score 0.454, right 26.8% Before: 70 answers, score 0.549, right 32.9% Before: 74 answers, score 0.648, right 36.5% Before: 77 answers, score 0.750, right 42.9% Before: 121 answers, score 0.856, right 47.1% Before: 809 answers, score 0.9855, right 70.1% After: 82 answers, score 0.271, right 23.2% After: 217 answers, score 0.351, right 39.2% After: 199 answers, score 0.447, right 51.3% After: 166 answers, score 0.548, right 54.8% After: 139 answers, score 0.656, right 58.3% After: 140 answers, score 0.749, right 77.1% After: 169 answers, score 0.851, right 79.9% After: 109 answers, score 0.933, right 95.4% Softmax over A–E, as is calibration error 0.286 After temperature T ≈ 3.80 calibration error 0.041, same questions Largest dot: 809 answers scored 0.9 to 1.0 and were right 70.1% of the time. Dashed line: score equals accuracy. Circle size grows with answers in the bin. Calibration errors: my arithmetic from his tables.
Read as probabilities, the masked scores run far ahead of how often the model is right. One global temperature pulls them onto the diagonal without changing a single answer.

Two details in the second table matter. Accuracy is still 725 of 1,221, as it has to be: dividing every logit by the same positive number cannot change which one is largest. And since those 725 are the base model’s correct answers, the temperature was evidently fitted and checked on the same 1,221 questions. The post doesn’t mention a separate calibration split. One parameter leaves little room to overfit, but 0.041 is probably an in-sample number.

What a softmax over five letters measures

What I wanted to know next is what 0.9988 is a probability of. Write \(p(y)\) for the model’s probability of token \(y\) over its whole vocabulary, which for Qwen3-1.7B means a logit vector of 151,936 entries according to its published configuration, and \(m\) for the total probability the five letters receive. Masking and renormalising gives

\[p_{\text{mask}}(y) = \frac{p(y)}{m}\]

That is a conditional probability: the chance that the next token is B, given that it is one of A to E. The ratios between the allowed letters survive untouched, so the ranking and the answer are exactly the ones the model already had. That part of the mask costs nothing. What it loses is \(m\), and \(m\) changes from question to question. If the five letters hold 95% of the model’s probability, a raw 0.9489 on B comes out as 0.9988. If they hold 5%, a raw 0.04994 on B also comes out as 0.9988. The two outputs are identical. In the second case the model mostly wanted to emit something else, whether a filler word, a line break or the word “Cave” itself, and the mask tidied that away. A raw 0.30 on B becomes 0.33 when the letters hold 90% of the mass and 0.60 when they hold half.

Two different models of the next token, one masked score Illustration with invented values, not measurements. Two bars show the full next-token probability. In the first case the five option letters hold 95 percent of it, with B at 94.89 percent and every other token at 5 percent. In the second case the letters hold 5 percent, with B at 4.994 percent and every other token at 95 percent. After masking to the five letters and renormalising, B scores 0.9988 in both cases. FULL NEXT-TOKEN PROBABILITY · ILLUSTRATION · BAR = 100% OF THE VOCABULARY The letters hold 95% The letters hold 5% B: 94.89% of the full probability (illustration) B: 4.994% of the full probability (illustration) Every token other than the five letters: 5% (illustration) Every token other than the five letters: 95% (illustration) B 94.89% every other token, 95% every other token, 5% B 4.994% B after mask: 0.9988 B after mask: 0.9988 Invented values. In both cases B holds 99.88% of whatever the five letters receive, so the masked score is the same. The letters' share, m, is what the mask discards.
The mask keeps the ranking among the letters and drops how much of the model's probability the letters received at all.

Temperature scaling works on the five logits that survive the mask, so it cannot tell those two cases apart. The information that separates them was gone before it ran. It can pull the average score down to the average accuracy, which is what Tahir’s second table shows. What it cannot do is give a lower score to the particular answer the model was least committed to.

There is an older warning about reading answers off the first token. In February 2024 Xinpeng Wang and colleagues compared multiple-choice scores taken from first-token probabilities with the answers instruction-tuned models actually wrote out, and found mismatch rates reaching over 60%; the misalignment persisted when prompts were constrained to start with an option letter (Wang et al., 2024). A mask doesn’t remove that gap. It removes the evidence of it.

Putting a number on the lost mass

Sanket Badhe, Priyanka Tiwari and Deep Shah named the problem in May. They call it renormalization bias: mass the model gives to synonyms of a label is dropped, and the distribution that remains is, in their words, “artificially peaked” (Badhe et al., 2026). Their remedy, which they call Semantic Softmax, credits each label with the probability on a neighbourhood of related tokens instead of discarding it. On Qwen-3-1.7B, the same size and family as Tahir’s model, the expected calibration error on GoEmotions falls from 0.574 to 0.069 and on CivilComments from 0.482 to 0.108. On Phi-4-mini the falls are 0.421 to 0.065 and 0.395 to 0.092, and AUROC and F1 rise in all four cases. The paper’s text gives 0.071 for the Qwen GoEmotions result where its table says 0.069; I use the table.

Calibration error of masked scores, before and after a fix Dumbbell chart of expected calibration error. Badhe and colleagues, standard masked softmax versus Semantic Softmax: Qwen-3-1.7B on GoEmotions 0.574 to 0.069; Qwen-3-1.7B on CivilComments 0.482 to 0.108; Phi-4-mini on GoEmotions 0.421 to 0.065; Phi-4-mini on CivilComments 0.395 to 0.092. Below a separator, Tahir's Qwen3-1.7B on CommonsenseQA, masked softmax versus temperature scaling: 0.286 to 0.041, the writer's arithmetic from his tables, in-sample. EXPECTED CALIBRATION ERROR · LOWER IS BETTER · BADHE ET AL. MAY 2026; TAHIR OCT 2026 Masked softmax With the paper's fix (rows 1–4) or temperature (row 5) Qwen-3-1.7B · GoEmotions Qwen-3-1.7B · CivilComments Phi-4-mini · GoEmotions Phi-4-mini · CivilComments Qwen3-1.7B · CommonsenseQA 0.574 standard, Qwen-3-1.7B GoEmotions, Badhe et al. Table 1 0.482 standard, Qwen-3-1.7B CivilComments, Badhe et al. Table 1 0.421 standard, Phi-4-mini GoEmotions, Badhe et al. Table 1 0.395 standard, Phi-4-mini CivilComments, Badhe et al. Table 1 0.286 masked softmax, Tahir's bins, writer's arithmetic 0.069 Semantic Softmax, Qwen-3-1.7B GoEmotions 0.108 Semantic Softmax, Qwen-3-1.7B CivilComments 0.065 Semantic Softmax, Phi-4-mini GoEmotions 0.092 Semantic Softmax, Phi-4-mini CivilComments 0.041 after temperature scaling, Tahir's bins, writer's arithmetic, in-sample 0.574 0.482 0.421 0.395 0.286 0.069 0.108 0.065 0.092 0.041 0 0.2 0.4 0.6 Rows 1–4: Badhe et al., Table 1, emotion and toxicity labels. Row 5: my arithmetic from Tahir's bin tables, multiple choice, fitted and scored on the same questions. Different tasks; read the rows separately.
Every masked baseline here is badly calibrated, and each fix recovers most of the gap. The paper's fix works by putting the discarded probability back.

The caveats are ordinary ones: two small models, emotion and toxicity labels rather than multiple choice, and the authors testing their own method. The paper shows that putting the discarded mass back repairs calibration. It doesn’t report \(m\) itself, and I found no 2026 paper, as of 17 October, that reports the share of probability falling outside the allowed label set as a number in its own right.

Asking the model to say how confident it is doesn’t get around the problem. In February Nariman Naderi and colleagues tested 48 language models on 300 board-style gastroenterology questions and found that only 5 reported their confidence better than random, with the best calibration error at 0.100 (Naderi et al., 2026). The best performers there were 2024-era systems such as o1-preview and GPT-4o, and it is one specialty, but it is no evidence that asking works better.

Which number the server hands back

The practical question is whether your stack gives you \(p(y)\) or \(p(y)/m\), and in vLLM the answer depends on how you ask for the restriction. I read the main branch on 17 October. In the GPU model runner, the structured-output bitmask is applied to the logits before the sampler runs (vLLM, gpu_model_runner.py), and the bitmask function it calls from XGrammar, one of its grammar backends, sets disallowed logits to negative infinity (XGrammar, matcher.py). The sampler’s docstring then lists its steps in order (vLLM, sampler.py). In the default mode, raw_logprobs, it computes logprobs first from the logits as they arrive. Only after that does it apply the allowed-token whitelist, the bad-words list, logit bias and penalties.

Where vLLM reads the logprobs it returns Pipeline in order: model logits over the full vocabulary, then the grammar bitmask used by guided decoding, then the point where the default raw logprobs are read, then the allowed-token-ids whitelist and bad words, then logit bias and penalties, then sampling with temperature and top-p. A grammar restriction is applied before the read, so returned logprobs are already renormalised over the grammar's tokens. Restrictions applied after the read leave the returned values at their full-vocabulary level. Based on reading vLLM's main branch on 17 October 2026, not on a run. ORDER OF OPERATIONS ON ONE DECODING STEP · VLLM MAIN BRANCH, READ 17 OCT 2026 Model logits Grammar mask Allowed tokens Logit bias Sample full vocabulary guided decoding bad words penalties temperature, top-p Returned logprobs read here Before the read: logprobs come back already renormalised over the grammar After the read: returned logprobs keep their full-vocabulary values, so m can be summed Default mode raw_logprobs. From the source and its docstring, not a test run.
The same five-token restriction gives back two different numbers depending on which side of the read it is applied.

If the code does what it says, the consequence is this. Restrict vLLM to five choices through a grammar and the logprobs you get back are already renormalised over what the grammar allowed, so \(m\) is gone. Restrict it to the same five tokens with allowed_token_ids and the logprobs come back at their full-vocabulary values, and you can sum \(m\) yourself, provided you ask for the logprobs of all five letters (with logprob_token_ids, or logprobs=-1 if the engine’s max_logprobs is raised to the vocabulary size), since a top-k list can leave some out, most likely when \(m\) is small. That is my reading of the code path; I have not run it, and a stack that relies on the difference should test it. OpenAI’s API reference says logit bias is added to the logits “prior to sampling,” that values of −100 or 100 should amount to a ban or exclusive selection, and that top_logprobs returns at most 20 alternatives (OpenAI, 2026). It does not say whether the logprobs it returns are taken before or after the bias. If you build a classifier on it, test that before anything else.

The mask also moves the answers

So far the mask has been harmless to accuracy, because one forward pass over five letters keeps the model’s own ranking. When the constraint runs across many decoding steps, the effect on accuracy is no longer neutral. Avinash Reddy and colleagues call it a “projection tax”: masking and renormalising at every step can push generation down paths that are valid but wrong when the model gives little probability to the valid continuations. Letting the model draft without constraint and then constraining the result improved strict structured accuracy by up to 24 points, from 15.2% to 39.0% on GSM8K with a 1B model (Reddy et al., 2026). Janghoon Lee’s study of tool-calling routers shows how easy it is to blame the wrong part. Compared with free generation, a grammar over tool names combined with a stop at the first line cut accuracy on declining a call, the right answer when no tool applies, in four of six model-and-language cells (intervals excluding zero), by up to 29.5 points, and raised it significantly in none. On items that needed a tool, the combination helped throughout. Pulling the two apart changes the picture: on the smallest model in Korean, the stop alone cost 20.0 points and the grammar restriction gave back 19.5. Across all cells, of the 698 declines the restriction repaired, 545 had been outputs with no readable answer at all (Lee, 2026). In October, Vineet Kumar and colleagues found that structured formats such as JSON or XML that put the answer field before the reasoning cause steep accuracy drops, particularly in smaller models, while reasoning-first ordering matches or exceeds free-form output (Kumar et al., 2026).

None of this makes the decision-model trick a bad idea. For a fixed set of options it is fast and cheap, and the argmax is a fair answer. Calibration is a separate property, and it has to come from training or fitting; the mask doesn’t supply it. When I looked at Jev in September, an outside test, on one small set of letters, found its stated probabilities underconfident at both ends, the opposite direction from the overconfidence of a raw softmax over five letters.

The cheapest improvement is one extra number. Before the mask, add up the probability the allowed tokens received and log it next to the answer. A high masked score with a small \(m\) is the model telling you it wanted to say something you didn’t allow. That is information about the question, your options or your prompt format, and the renormalised score is built to hide it.

References

  1. Tahir, N. (2026). Build your own decision model. Another Dev’s Two Cents, 10 October 2026.
  2. Hacker News (2026). Build your own decision model. Discussion thread, 10 October 2026; 479 points when checked on 17 October 2026.
  3. Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review 78(1): 1–3, January 1950.
  4. Qwen (2026). Qwen3-1.7B configuration file. Hugging Face, accessed 17 October 2026.
  5. Wang, X., Ma, B., Hu, C., Weber-Genzel, L., Röttger, P., Kreuter, F., Hovy, D. and Plank, B. (2024). “My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models. arXiv:2402.14499, 22 February 2024.
  6. Badhe, S., Tiwari, P. and Shah, D. (2026). The Silent Vote: Improving Zero-Shot LLM Reliability by Aggregating Semantic Neighborhoods. arXiv:2605.09739v1, 10 May 2026.
  7. Naderi, N., Safavi-Naini, S. A. A., Savage, T., Khalafi, M. A., Lewis, P. R., Atf, Z., Nadkarni, G. and Soroush, A. (2026). Across generations, sizes, and types, large language models poorly report self-confidence in gastroenterology clinical reasoning tasks. npj Gut and Liver 3: 6, 5 February 2026.
  8. vLLM project (2026). vllm/v1/worker/gpu_model_runner.py and vllm/v1/sample/sampler.py. Source code, main branch, read 17 October 2026.
  9. XGrammar project (2026). python/xgrammar/matcher.py. Source code, apply_token_bitmask_inplace docstring, main branch, read 17 October 2026.
  10. OpenAI (2026). Create chat completion. API reference, accessed 17 October 2026.
  11. Reddy, A., Walker, T. T., Ide, J. S. and Bedi, A. S. (2026). The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding. arXiv:2603.03305, 8 February 2026.
  12. Lee, J. (2026). Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention. arXiv:2608.13959, 14 August 2026.
  13. Kumar, V., Kanishka and Mandora, B. (2026). Structure Tax: How Structured Output affects LLMs Performance. arXiv:2610.12056, 8 October 2026.