Four Executions per Call, None Delivered

Every LLM client library retries. The OpenAI Python SDK ships with DEFAULT_MAX_RETRIES = 2. When a call times out, the client closes the connection and sends the same prompt again. Most of the time that is the right behavior. A paper posted to arXiv on 5 November, “Cancellation Is Not Enough” by Xindan Zhang (University of Toronto) and Fei Wang (UBC Okanagan), measures what happens when it isn’t: a vLLM engine that gets a short burst of overload and then cannot climb out of it, even after traffic drops well below what it can serve.

The usual answer to “the engine keeps working on requests nobody is waiting for” is to propagate the client’s disconnect and abort the work. The paper tests that answer directly and finds that it does not restore service in any run. I read the paper in full because the setup is unusually careful about what it counts, and because one of its side findings, about vLLM’s own metrics, matters to anyone who autoscales on them.

The setup

Everything runs on one NVIDIA H100 80GB with vLLM 0.30.0 from the official container, serving three open models: Qwen2.5-7B-Instruct, Qwen3-14B and Llama-3.1-8B-Instruct. Prompts are random token IDs, 12,000 ± 1,200 tokens long, all sharing a 2,000-token prefix. Response lengths are lognormal with a median of 300 tokens. The authors point out that the mean prompt and response lengths sit close to a published real-world conversation trace.

The client gives up on an attempt after 120 seconds and retries up to three times, each after a random wait of 0.25 to 3 seconds. That is one more retry than the OpenAI SDK default. Every attempt goes through a proxy the authors wrote, which can propagate or withhold cancellation and can budget or coalesce retries.

Load is set in multiples of each model’s measured capacity, written Ĉ. For Qwen2.5-7B-Instruct, Ĉ is 2.78 calls per second. Capacity as a single number is a simplification. Work at HPEC 2025 modeled an inference server’s request capacity as a saturation function of arrival rate. Here it is calibrated once per engine build, by raising the rate in stages until the waiting queue starts to grow. Each run has three phases: 120 seconds at 0.6 Ĉ, a 107-second pulse at 2.7 Ĉ, then a 720-second “hold” back at 0.6 Ĉ. A call counts as delivered if the client gets its full response within 840 seconds of when it was planned to arrive.

The pulse is sized on purpose. Its excess over capacity leaves a backlog that would take the engine about 182 seconds to clear at full speed, 1.5 times the client’s deadline. So a call queued at the end of the pulse can’t make its deadline even in the best case. Without retries, the spare 0.4 Ĉ during the hold drains that backlog in about 455 seconds. Service should come back in minutes. The paper asks whether it comes back at all.

Retries alone: zero

With retries on and cancellation off, Qwen2.5-7B-Instruct delivered none of the calls that arrived during the hold. With both off, it delivered 72.4 to 74.8% of them and recovered 480 to 540 seconds after the pulse, shortly after the backlog drained. Same engine, same trace, same load. The only difference is whether the client sends the call again.

The arithmetic is in the paper and it is short. Calls arriving at 0.6 Ĉ that each make four attempts offer 2.4 Ĉ of work, more than twice what the engine can finish. The measurements match. Over the hold window they tracked, vLLM accepted executions at 2.33 to 2.47 times Ĉ and completed them at 95.1 to 97.6% of Ĉ. The engine was running flat out. Retries created 75.7 to 76.4% of the executions, and every one of those executions finished for a client that had already left.

This is a textbook metastable failure. Bronson and colleagues, who named the pattern in 2021, describe failures that “are much easier to explain in hindsight than to predict.” The trigger, the pulse, is gone. What keeps the system down is a loop the trigger started: late answers cause timeouts, timeouts cause retries, retries cause more late answers. A follow-up study of public incident reports found 22 such failures across 11 organizations and counted at least 4 of 15 major AWS outages in a decade among them.

Work accepted versus work finished during the hold Qwen2.5-7B-Instruct on vLLM, measured over 347 to 947 seconds of each run, in multiples of the engine's capacity. Calls arrived at 0.6 times capacity. With retries on and cancellation off, vLLM accepted executions at 2.33 to 2.47 times capacity and completed them at 0.95 to 0.98 times capacity. With retries on and cancellation on, it accepted executions at 2.35 to 2.50 times capacity and completed them at 0.002 to 0.009 times capacity. EXECUTIONS PER SECOND · HOLD WINDOW 347–947 S · QWEN2.5-7B · MULTIPLES OF CAPACITY Capacity (1.0 Ĉ) Calls arriving Accepted, cancellation off Completed, cancellation off Accepted, cancellation on Completed, cancellation on 0.6 Ĉ arrivals during the hold — Zhang and Wang, 5 Nov 2026 2.33–2.47 Ĉ accepted, retries on, cancellation off — Zhang and Wang, 5 Nov 2026 0.951–0.976 Ĉ completed, retries on, cancellation off — Zhang and Wang, 5 Nov 2026 2.35–2.50 Ĉ accepted, retries on, cancellation on — Zhang and Wang, 5 Nov 2026 0.002–0.009 Ĉ completed, retries on, cancellation on — Zhang and Wang, 5 Nov 2026 0.6 Ĉ 2.33–2.47 Ĉ 0.95–0.98 Ĉ 2.35–2.50 Ĉ 0.002–0.009 Ĉ Retries on in all four lower rows. Solid bar to the lowest run, line to the highest. Source: Zhang and Wang, arXiv:2611.08684.
With cancellation off, the engine runs at capacity for nobody. With cancellation on, it accepts the same flood and finishes almost nothing.

Cancellation: thirteen to fifteen percent

So abort the orphaned work. The proxy closes its connection to vLLM when the client leaves, and vLLM aborts the execution. In the traced runs, every recorded abort reached vLLM within half a second of the client’s departure. The mechanism works as designed.

It barely helped. With retries and cancellation both on, Qwen2.5-7B-Instruct delivered 13.3 to 15.2% of the hold’s calls, most of them after the hold had ended. vLLM still accepted executions at 2.35 to 2.50 times Ĉ, but now it completed them at 0.2 to 0.9% of Ĉ. The other two models did a little better, 28.1 to 37.7%, and also failed to recover. Without retries, cancellation was a clear win: 89.0 to 91.4% delivered and recovery 360 seconds after the pulse, faster than without it.

The traces show why. The retries that vLLM scheduled at all were first scheduled a median of 120.0 to 121.3 seconds after their predecessor was aborted, and they began their first step a median of 0.22 to 0.47 seconds before their own 120-second deadline. The waiting queue held about one deadline’s worth of attempts. A retry joined the back of it, waited out almost its whole timeout, got onto the GPU just as the client was about to give up, and was aborted. Then the next retry joined the back. The mean waiting queue came to 98.2 to 99.2% of what Little’s law predicts if every execution waits out its full deadline.

Cancellation shortens the work. It doesn’t stop the retry from creating new work, and in this loop the creation is the problem.

Where the prompt work went

The tracing block follows each execution through vLLM’s scheduler and attributes every computed prompt token, meaning one computed rather than pulled from the prefix cache. With cancellation on, executions that were later aborted took 83.8 to 85.2% of the computed prompt tokens at vLLM’s default of 8,192 batched tokens per step. Most of the engine’s prefill went to clients that had left or were about to.

Part of that couldn’t be stopped. vLLM 0.30.0 schedules asynchronously, so up to two steps of an execution may already be issued to the GPU when the abort comes, and those run to completion. At 8,192 tokens per step, that in-flight work was 51.7 to 55.1% of what the aborted executions computed. Shrinking steps to 2,048 tokens cut it to 23.4 to 27.0% and raised hold-call delivery from 13.2 to 15.8% up to 21.5 to 24.3%. It still didn’t restore service. Prefix caching didn’t save anything either: apart from the shared 2,000-token prefix, no retry reused its predecessor’s prefill.

Where prefill went with cancellation on Traced runs of Qwen2.5-7B-Instruct with retries and cancellation on. At 8,192 batched tokens per step, executions later aborted held 83.8 to 85.2 percent of computed prompt tokens, 51.7 to 55.1 percent of the aborted work was already in flight when the abort came, and executions aborted before their first output, which vLLM's prompt-token counter does not record, held 47.3 to 50.3 percent of computed prompt tokens. At 2,048 tokens per step the same figures were 79.4 to 80.7, 23.4 to 27.0, and 33.6 to 38.0 percent. COMPUTED PROMPT TOKENS · QWEN2.5-7B TRACED RUNS · RETRIES AND CANCELLATION ON 8,192 tokens per step (vLLM default) 2,048 tokens per step Held by executions later aborted In flight at the abort (of aborted work) Missed by the prompt-token counter Held by executions later aborted In flight at the abort (of aborted work) Missed by the prompt-token counter 83.8–85.2% — Zhang and Wang, 5 Nov 2026 51.7–55.1% of aborted executions' computed prompt tokens — Zhang and Wang, 5 Nov 2026 47.3–50.3% — Zhang and Wang, 5 Nov 2026 79.4–80.7% — Zhang and Wang, 5 Nov 2026 23.4–27.0% of aborted executions' computed prompt tokens — Zhang and Wang, 5 Nov 2026 33.6–38.0% — Zhang and Wang, 5 Nov 2026 83.8–85.2% 51.7–55.1% 47.3–50.3% 79.4–80.7% 23.4–27.0% 33.6–38.0% First and third rows are shares of all computed prompt tokens; the second is a share of the aborted executions' tokens. Three runs per condition; bar to the lowest, line to the highest. Source: Zhang and Wang, arXiv:2611.08684.
Smaller steps cut the work that can't be stopped by about half. Most of the prefill still goes to executions that get aborted.

The counter that misses the problem

Of everything in the paper, this is what I’d send to people running vLLM in production. The vllm:prompt_tokens_total counter records a prompt only when its execution produces its first output token. vLLM’s metrics design page says so: the frontend collects “the total number of prompt tokens processed by the prefills that completed in this iteration.” A prefill that gets aborted before it completes is never counted.

Under retries plus cancellation, that is most of the waste. Across a whole traced run, executions aborted before their first output held 47.3 to 50.3% of computed prompt tokens at 8,192 per step and 33.6 to 38.0% at 2,048. In the hold’s last 240 seconds, at the default step size, they held 86.3 to 93.7%. You don’t need the authors’ tracing to see the gap. vLLM also exports vllm:prefix_cache_queries, which counts each prompt’s full length once at admission. In the diagnostic runs on Qwen2.5-7B-Instruct, the prompt-token counter’s rise over the hold fell short of the admission counter’s by 63.1 to 67.9%. On the other two models the shortfall was 23.4 to 48.3% and 21.4%.

So a dashboard or autoscaler reading the prompt-token counter during a retry storm would see the engine doing much less prefill than it is, at the moment it is doing almost nothing else. Comparing the two counters is a cheap alarm, and the paper notes that none of its own controls read either one.

What brought service back

The authors then test controls that keep retries out of vLLM’s queue. There are two families: shed the retry, or attach it to the execution its call already has running.

A retry budget sheds. The proxy answers a retry with HTTP 429 unless retries stay within 10% of first attempts over a sliding 60-second window. That ratio comes from the Google SRE book, which describes a per-client retry budget where “a request will only be retried as long as this ratio is below 10%.” On Qwen2.5-7B-Instruct the budget refused 93.4 to 94.7% of retries, restored service, and delivered 77.5 to 82.0% of the later hold calls within 120 seconds. But it delivered only 51.0 to 54.4% of the pulse calls by 840 seconds, against 50.9 to 53.2% with no control at all. For those calls the budget did nothing.

Coalescing attaches. The proxy keys each call by its identifier and full generation parameters, and when a retry arrives while the call’s execution is still running, the retry becomes a second listener on that execution instead of a new one. It attached 99.0 to 99.5% of retries, delivered 83.0 to 89.5% of the later hold calls within 120 seconds, more than the budget, and delivered every pulse call by 840 seconds, nearly half of them late. On the other two models both controls also restored service, and only coalescing rescued the pulse calls.

Combined with coalescing, cancellation goes from unhelpful to harmful. Pair coalescing with immediate cancellation and the proxy kills each execution the moment the client leaves, half a second before the retry shows up to attach to it. In those runs the proxy attached no retries, vLLM aborted 6,193 and 6,296 executions, and service did not recover. Defer cancellation by 10 seconds, longer than the client’s 3-second maximum backoff, and coalescing attached 99.6% of retries and delivered every pulse and hold call. In those runs the proxy never had to cancel anything.

Pulse calls delivered under each overload control Share of calls planned during the 2.7-times-capacity pulse that were delivered within 840 seconds, Qwen2.5-7B-Instruct. No control 50.9 to 53.2 percent. Retry budget 51.0 to 54.4 percent. vLLM queue cap of 100, 40.7 to 40.9 percent. Proxy cap of 100 with newest-first dequeue and cancellation, 93.3 to 93.8 percent. Coalescing, 100 percent. Coalescing with cancellation deferred by 10 seconds, 100 percent. PULSE CALLS DELIVERED BY 840 S · QWEN2.5-7B · SHED VS ATTACH SHED OR NOTHING ATTACH OR SERVE NEWEST No control Retry budget (10%) vLLM queue cap, Q = 100 Proxy cap, newest first Coalescing Coalescing, cancel after 10 s 50.9–53.2% — composition block, Zhang and Wang, 5 Nov 2026 51.0–54.4% — Zhang and Wang, 5 Nov 2026 40.7–40.9% — Zhang and Wang, 5 Nov 2026 93.3–93.8% — Zhang and Wang, 5 Nov 2026 100%, every pulse call — Zhang and Wang, 5 Nov 2026 100%, every pulse call — Zhang and Wang, 5 Nov 2026 50.9–53.2% 51.0–54.4% 40.7–40.9% 93.3–93.8% 100% 100%
Shedding brings the engine back but abandons the calls caught in the burst. Attaching delivers all of them, some late. Each control is compared with its own block's no-control runs in the paper; the first row is the composition block's.

Queue caps sat in between. vLLM’s own cap at 100 queued executions restored service but failed 477 calls per run and delivered 40.7 to 40.9% of the pulse calls, fewer than its own no-control runs. A cap of 100 at the proxy restored service only when the proxy served the newest executions first, which delivered 93.3 to 93.8% of pulse calls. Serving the newest first works for the reason the retry timing suggests: the oldest attempt in the queue belongs to a client that is about to leave.

Across all 137 runs there’s one pattern the authors are careful not to turn into a threshold. Every run that recovered created executions at the proxy below Ĉ. One run at 0.98 Ĉ still didn’t. Below capacity was necessary in every case they tried and not sufficient.

What I’d take from it

The headline result reads like a vLLM bug and isn’t one. Cancellation did what it promised, within half a second, every time. The trouble is that it acts on executions, and the overload is driven by attempts. A retry arrives as a new execution no matter how quickly its predecessor was cleaned up. The paper’s first rule follows from that: track retry load by the executions you create, not by the requests you see.

The second rule is the one I hadn’t thought about. If every call needs an answer, which is the agent case where a task can’t finish without that one step, attach retries to running work and keep that work alive long enough for the retry to find it. This needs every attempt of a call to carry the same identifier. The authors checked the OpenAI Python SDK 2.54.0 and found it marked each retry with a retry count but sent no per-call key. A header the caller added was resent with every retry, so the fix is available, but it’s on the caller. Behind a load balancer with several proxies, coalescing also needs shared state, which this testbed didn’t have.

The limits are stated plainly: one vLLM version, one GPU type, synthetic single-turn prompts, an open-loop client with one deadline and one retry policy. The collapse appeared only after pulses whose backlog came close to or exceeded the client’s deadline. With retries on, every model recovered after a 1.5 Ĉ pulse and none after 2.7 Ĉ. A client with the SDK’s default 600-second timeout would need roughly a ten-minute backlog before the same loop could start, and the authors say so. That timeout, though, bounds each wait for the next byte, not the whole call, so an agent framework that sets its own wall-clock limit per step is closer to the paper’s client than to the SDK default.

The metrics finding carries fewer caveats, because it follows from a documented rule and showed up on all three models. If you serve with vLLM and retry anywhere in front of it, put prompt_tokens_total and prefix_cache_queries on the same panel. When they diverge, the engine is busy prefilling for clients that have already gone. It’s the same lesson as the Cloudflare incident I wrote about last month: the number a serving stack reports is defined by when it gets recorded, and under failure that definition can hide most of what is going on.

References

  1. Zhang, X. and Wang, F. (2026). Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine. arXiv:2611.08684, submitted 5 November 2026.
  2. OpenAI (2026). openai-python, src/openai/_constants.py. GitHub, main branch, accessed 9 November 2026.
  3. The vLLM Team (2026). Metrics. vLLM design documentation, accessed 9 November 2026.
  4. Bronson, N., Aghayev, A., Charapko, A. and Zhu, T. (2021). Metastable failures in distributed systems. Proceedings of the Workshop on Hot Topics in Operating Systems (HotOS ‘21), pp. 221-227, June 2021.
  5. Huang, L., Magnusson, M., Muralikrishna, A. B., Estyak, S., Isaacs, R., Aghayev, A., Zhu, T. and Charapko, A. (2022). Metastable Failures in the Wild. 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI ‘22), July 2022.
  6. Forero Cuervo, A. (2016). Handling Overload. In Beyer, B., Jones, C., Petoff, J. and Murphy, N. R. (eds.), Site Reliability Engineering: How Google Runs Production Systems, ch. 21. O’Reilly Media, 2016.
  7. Burrill, D. J. et al. (2025). Predicting LLM Inference Server Request Capacity. 2025 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1-6, September 2025.