Nineteen of Eighty-Three: The Kernel Passes, and Still Changes Its Answer

A paper posted to arXiv on 27 October by Youwei Bao, Chengpeng Wang and Simin Chen, of the STAR Lab at George Mason University and the National University of Singapore, is called “God Does Not Play Dice: Enforcing Batch Invariance in Agentic Kernel Generation.” Behind the title is a plain audit. Agents are now writing GPU kernels and the good ones get released. The authors went and checked those released kernels for a property nobody was grading them on, and found that about one in four of the applicable ones fails it.

The property is batch invariance. If my request and your request land in the same batch on a serving GPU, my output should not depend on the fact that you were there. Serving systems batch continuously and regroup requests between runs, so the same prompt meets a different crowd every time. If the kernel’s arithmetic depends on the crowd, the same prompt at temperature 0 can produce a different answer on Tuesday than it did on Monday, with the same weights and the same sampling settings.

I have written before about what a scoreboard does not record: about one anti-cheating rule in three going missing from a compaction summary while the benchmark score stays flat, and about an oracle that compares bytes rather than behaviour. This is the same shape of problem in a much narrower place. Kernel agents are scored on two things, correctness against a reference within a tolerance and speed, and a kernel can hold both while quietly failing a third.

Why a correct kernel can be wrong

Floating-point addition is not associative. Change the order of a sum and you can change the last bits of its result. A GPU kernel that reduces a long vector decides that order at launch time, from the shape of the work it was handed: how many splits to cut the reduction into, what tile size to use, which of several implementations to dispatch to. Change the batch size or composition and those decisions change, so the reduction order changes, so the bits change, for a sample that was itself untouched.

The paper is careful to separate this from the neighbouring problems. Run-to-run nondeterminism from unordered atomics is a different thing, and the authors note such atomics “are uncommon in modern LLM forward inference.” Differences from changing hardware or library versions are a third thing. Batch dependence is the case where the input and the environment are both fixed and only the surrounding batch moves.

This is not a newly noticed hazard in numerical computing. Collange, Defour, Graillat and Iakymchuk, writing in Parallel Computing in 2015, open with the same observation: on multi-core and many-core architectures floating-point computations, “especially reductions, may become non-deterministic and, therefore, non-reproducible mainly due to the non-associativity of floating-point operations” (10.1016/j.parco.2015.09.001). Their answer was a long accumulator that makes the sum correctly rounded by construction, and in a companion paper Defour and Collange separate the two repairs available: “work reassignment,” which forces an execution order, and fixed-point accumulation, which removes the rounding error that made order matter in the first place (10.15439/2015F86). The hardware is a decade newer; the arithmetic is not.

What is new is who is writing the kernels. A human author who knows this hazard avoids it by habit. An agent optimizing against a reward of correctness-plus-speed has an active incentive to reach for exactly the shape-dependent tricks that break it, because they are fast and they still pass the tolerance check.

The audit

The authors collected 131 released agent-generated kernel artifacts, kept the 83 that have a meaningful semantic batch dimension, and tested those for violations. Nineteen failed, 22.9 percent of the applicable set.

Released agent-generated kernel artifacts, narrowed to those with confirmed batch-invariance violations A bar chart on a 0 to 131 scale. Of 131 released artifacts, 83 have a meaningful semantic batch dimension, and 19 of those 83 contain a confirmed batch-invariance violation, leaving 64 that pass. By source, KernelBench.com contributes 79 batch-applicable of 126 released with 18 violations and a 77.2 percent pass rate; K-Search on FlashInfer-Bench contributes 4 of 5 released with 1 violation and a 75.0 percent pass rate. RELEASED KERNEL ARTIFACTS AUDITED · BAO, WANG AND CHEN, ARXIV 2610.36623, TABLE 1 Released artifacts With a batch dimension Confirmed BI violations 131 released artifacts — Table 1, arXiv 2610.36623 83 batch-applicable artifacts — Table 1, arXiv 2610.36623 19 confirmed batch-invariance violations — Table 1, arXiv 2610.36623 131 83 19 By source, of the batch-applicable artifacts: KernelBench.com K-Search on FlashInfer-Bench 79 of 126 released 4 of 5 released 18 violations 1 violation 77.2% pass 75.0% pass Artifacts without a meaningful batch dimension are excluded; the property does not apply to them.
Every one of these kernels was released as a success. The audit asks a question their acceptance criteria never did.

Most of the sample comes from one place, KernelBench.com, which contributes 79 of the 83 and 18 of the 19 failures. That is a concentration worth saying out loud: this is largely an audit of one leaderboard’s published artifacts, not a survey of the field. The second source, K-Search’s entries on FlashInfer-Bench, offers four applicable kernels and one failure, which is too few to carry a rate of its own.

The gate

The tool the paper proposes, BIEvolve, is deliberately unambitious in scope. It does not write kernels. It bolts onto an existing kernel agent as two acceptance gates in that agent’s own optimization loop, placed before the performance benchmark so that a batch-dependent candidate never gets timed, and it hands the agent a diagnostic when it rejects something.

The first gate is static. A small domain-specific language declares, per kernel family, which symbols are batch sources and which parameters must not depend on them. Three parameters are protected in the current language: NUM_SPLITS, the split and reduction structure; RECURRENCE_SEGMENTS, how a recurrence is decomposed; and RNG_SEED. The analyzer walks the generated Python, Triton and CUDA for any data or control dependence from a batch symbol to one of those, including through aliases: split_k, num_partitions, max_splits and a dozen more spellings of the same idea. The second gate is dynamic. It runs the candidate, holds one sample fixed, changes the batch around it, and compares the outputs bit for bit.

The division of labour between the two is the part I find most useful, because it is measured rather than asserted. On a shared pool of 200 independently labelled candidates, the static analyzer catches 83 percent of confirmed violations and the dynamic validator 55 percent; together they catch 92. They overlap on 46 percent, and each finds something the other misses, 37 percent unique to static and 9 percent unique to dynamic.

Violation detection, false rejection and evaluation cost for each gate and the pair A horizontal bar chart on a 0 to 100 percent scale over a shared pool of 200 candidates. The static analysis module detects 83 percent of confirmed violations, falsely rejects 2 percent of clean candidates and takes 7.69 milliseconds per candidate. The dynamic validation module detects 55 percent, falsely rejects none and takes 2.24 milliseconds. The two together detect 92 percent, falsely reject 2 percent and take 10.21 milliseconds. VIOLATION DETECTION ON A SHARED POOL OF 200 CANDIDATES · ARXIV 2610.36623, TABLE 3 false reject ms each Static analysis only Dynamic validation only Both gates 83% of confirmed violations detected — Table 3, arXiv 2610.36623 55% of confirmed violations detected — Table 3, arXiv 2610.36623 92% of confirmed violations detected — Table 3, arXiv 2610.36623 83% 55% 92% 2% 7.69 0% 2.24 2% 10.21 0 25 50 75 100 The two gates overlap on 46% of violations; 37% are found only by static analysis and 9% only by the dynamic check. Milliseconds are medians per candidate. The paired figure preserves sequential early exit, so it is not the sum.
Neither gate is sufficient alone. Static analysis sees the dependency in the source; the dynamic check sees behaviour the rules were never written for.

I read the 9 percent as the honest part of the design. The static rules can only forbid what someone wrote down, and the paper says so: dependencies that cannot be traced to a declared batch symbol are “outside the current static-analysis scope.” Opaque dispatch inside a library, expert routing in a mixture-of-experts layer, an autotuner picking a different implementation: none of that is visible in the generated source. Running the thing twice is what catches those.

What enforcement costs

The experiment reruns three kernel agents with and without the gates: K-Search, and the Iterative Agent and Evolve Agent from the FlashInfer team. These are three different search strategies, namely single-trajectory refinement, a pool of elite candidates, and a planner with a separate world model. Each gets the same turn budget in both conditions, 20 turns for K-Search and 25 for the other two, with Gemini 3.7 Flash driving all three on one H100 NVL.

Batch-invariance pass rate before and after enforcement, by agent framework A dumbbell chart on a 0 to 100 percent scale. K-Search goes from a 75.7 percent batch-invariance pass rate at baseline to 97.4 percent with enforcement, retaining 86.4 percent of its speedup. The Iterative Agent goes from 48.6 to 89.2 percent, retaining 87.7 percent. The Evolve Agent goes from 78.4 to 89.2 percent, retaining 102.0 percent. Overall the pass rate goes from 67.6 to 91.9 percent, retaining 91.8 percent of the geometric mean speedup. Hollow dots are the baseline, filled dots the enforced rerun. BI PASS RATE, BASELINE VS ENFORCED · ARXIV 2610.36623, TABLE 2 speedup kept K-Search Iterative Agent Evolve Agent Overall 75.7% BI pass at baseline, K-Search — Table 2, arXiv 2610.36623 48.6% BI pass at baseline, Iterative Agent — Table 2, arXiv 2610.36623 78.4% BI pass at baseline, Evolve Agent — Table 2, arXiv 2610.36623 67.6% BI pass at baseline, overall — Table 2, arXiv 2610.36623 97.4% BI pass with enforcement, K-Search — Table 2, arXiv 2610.36623 89.2% BI pass with enforcement, Iterative Agent — Table 2, arXiv 2610.36623 89.2% BI pass with enforcement, Evolve Agent — Table 2, arXiv 2610.36623 91.9% BI pass with enforcement, overall — Table 2, arXiv 2610.36623 75.7 48.6 78.4 67.6 97.4 89.2 89.2 91.9 86.4% 87.7% 102.0% 91.8% 0 50 100 Hollow: baseline. Filled: enforced rerun, same turn budget. Driven by Gemini 3.7 Flash on one H100 NVL. Speedup kept is the enforced geometric mean over the baseline one, on the same tasks.
The weakest baseline improves most, and the agent that searches widest pays nothing, because it finds an invariant kernel that is just as fast.

Overall the pass rate goes from 67.6 to 91.9 percent while 91.8 percent of the baseline geometric mean speedup survives. The per-framework rows under that average do not behave alike.

The Iterative Agent starts worst, at 48.6 percent, and gains the most, to 89.2. It also needed the gates to fire on the most tasks, 43.6 percent against 23.1 for K-Search. A single-trajectory refiner that keeps pushing one implementation has the most room to wander into a shape-dependent optimization and the least machinery for backing out of it.

The Evolve Agent is the row that complicates the story about cost. It retains 102.0 percent of its speedup: the enforced rerun is marginally faster, 23.31x against 22.85x. The paper’s reading is that broad search can find an invariant alternative that is just as quick, and that the price of enforcement is partly a function of the search strategy rather than a fixed tax on the property. I would hold that number loosely, because it is one framework on one task suite, and a 2 percent gain sits well inside the noise that reruns of a stochastic search produce. It is still enough to show that batch invariance does not have to cost speed.

Where the cost does show up is exactly where you would predict. Splitting the tasks by whether the baseline kernel had a violation, retention is 86.0 percent on the tasks that did and 95.7 percent on the tasks that did not. The gates take away speed when the speed was coming from the batch-dependent trick.

Ordinary correctness, meanwhile, barely moves: 94.9 percent at baseline across all three frameworks, 95.7 percent with the gates on. The correctness number cannot see this problem, which is why it does not react when the problem is fixed.

The honest size of the claim

A few things keep this smaller than the abstract makes it sound.

The controlled experiment is one model on one GPU. Gemini 3.7 Flash wrote every kernel in every condition, on an H100 NVL. Whether a stronger model writes fewer batch-dependent kernels unprompted, or whether a different architecture shifts which optimizations are tempting, is untested here.

The task suite is chosen for risk, not coverage. It is drawn from the hard tasks on KernelBench.com and FlashInfer workloads, and weighted toward attention and linear attention precisely because those “commonly involve batch dependent partitioning, algorithm selection, and reductions.” The rates come from a sample picked for danger.

And a 91.9 percent pass rate is still roughly one kernel in twelve that remains batch-dependent after enforcement. The gates are a filter, not a proof.

The comparison the paper draws with existing deterministic-inference work is fair, and I checked it. Thinking Machines Lab’s September 2025 post on defeating nondeterminism in LLM inference ships a library that substitutes batch-invariant implementations for PyTorch operators through torch.Library; the paper notes it replaces “only four PyTorch operators.” The repository bears that out. enable_batch_invariant_mode registers exactly four: aten::mm, aten::addmm, aten::_log_softmax and aten::mean.dim. That is a demonstration that the property is achievable, not an answer for a serving stack full of agent-written kernels. The same post also puts a number on what the absence of the property feels like from outside: sampling a 235B-parameter model 1,000 times at temperature 0 on one fixed prompt produced 80 distinct completions, all identical for their first 102 tokens.

What I take from it

The audit is what I would keep, more than the tool. Nineteen of 83 released, correctness-passing kernels give a request a different answer depending on who else was in the batch, and nobody noticed because nobody was looking.

That is a general failure mode of optimizing agents, not a GPU fact. An agent pushed on two measured axes will find the parts of the space where an unmeasured third thing can be spent, and it will spend it, because spending it is rewarded. This particular property is cheap to test: hold a sample fixed, move the batch around it, compare the bits. That is why it can be gated at all, and it is more than most unmeasured properties offer.

References

  1. Youwei Bao, Chengpeng Wang and Simin Chen, “God Does Not Play Dice: Enforcing Batch Invariance in Agentic Kernel Generation,” arXiv:2610.36623, submitted 27 October 2026. https://arxiv.org/abs/2610.36623
  2. Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference,” 10 September 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
  3. thinking-machines-lab/batch_invariant_ops, batch_invariant_ops/batch_invariant_ops.py, read 28 October 2026. https://github.com/thinking-machines-lab/batch_invariant_ops
  4. S. Collange, D. Defour, S. Graillat and R. Iakymchuk, “Numerical reproducibility for the parallel reduction on multi- and many-core architectures,” Parallel Computing 49 (November 2015), 83–97. https://doi.org/10.1016/j.parco.2015.09.001
  5. D. Defour and S. Collange, “Reproducible floating-point atomic addition in data-parallel environment,” Proceedings of the 2015 Federated Conference on Computer Science and Information Systems, ACSIS vol. 5 (2015), 721–728. https://doi.org/10.15439/2015F86