Thirty-Six Tokens, Billed as 4,418

On 5 November six researchers at Queen’s University in Kingston, Ontario posted What the Middleman Delivers to arXiv. They opened accounts at 20 third-party API relays, the services that resell access to frontier models through one key, often at a discount, and sent each one the same requests they sent to the official APIs for GPT-6 Astra and Claude Opus 5. The first round ran from 2 to 9 October and the second on 20 October.

They checked three things a buyer cares about: whether the answers match the official model, whether the relay respects the tool permissions in the request, and whether the bill corresponds to anything you can verify. In all three, the relay owns the record you’d use to check it.

The answers

The accuracy test is built from questions the official model answers and cheaper models miss. From 96 candidates the authors kept 25 per target that the target got right at least twice in three tries and at least two of three cheaper substitutes (Claude Haiku 4.5, GPT-6 Luna and GLM-4.5-Flash) got wrong at least twice. Each relay and its official reference then answered every question five times. A pair is “Confirmed” if the Benjamini–Hochberg-adjusted q is under 0.05 and the accuracy gap is at least 0.20, and “Suspected” if the evidence is weaker.

In round one, 7 of 10 low-price relays had at least one flagged cell, against 2 of 10 relays priced at or above the official rate. That comparison gives p = 0.07, and the authors say plainly that it doesn’t clear 0.05 and that the sample wasn’t random. The gaps behind the flags are not subtle, though.

Official-minus-relay accuracy gap for Claude Opus 5, round one Dot plot of the round-one risk difference, official hit rate minus relay hit rate, for Claude Opus 5 at 20 relays from Figure 3 of arXiv 2611.07650. Low-price relays: R13 minus 0.032, R19 0.016, R15 0.017, R12 0.024, R04 0.080 (flagged), R08 0.152 (flagged), R17 0.256 (flagged), R01 0.288 (flagged), R11 0.574 (flagged), R20 0.591 (flagged). Same- or higher-price relays: R18 minus 0.008, R05 0.008, R02 0.016, R03 0.016, R09 0.016, R16 0.024, R07 0.032, R06 0.040, R14 0.088 (flagged), R10 0.192 (flagged). CLAUDE OPUS 5 · OFFICIAL MINUS RELAY HIT RATE · ROUND 1 · ARXIV 2611.07650 FIG. 3 0.20 gap for “Confirmed” −0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 Low price Same or higher price 10 relays 10 relays R13 −0.032 — Fig. 3 R19 0.016 — Fig. 3 R15 0.017 — Fig. 3 R12 0.024 — Fig. 3 R04 0.080, Suspected — Fig. 3 R08 0.152, Suspected — Fig. 3 R17 0.256, Confirmed — Fig. 3 R01 0.288, Confirmed — Fig. 3 R11 0.574, Confirmed — Fig. 3 R20 0.591, Confirmed — Fig. 3 R01 0.288 R17 0.256 R11 0.574 R20 0.591 R18 −0.008 — Fig. 3 R05 0.008 — Fig. 3 R02 0.016 — Fig. 3 R03 0.016 — Fig. 3 R09 0.016 — Fig. 3 R16 0.024 — Fig. 3 R07 0.032 — Fig. 3 R06 0.040 — Fig. 3 R14 0.088, Suspected — Fig. 3 R10 0.192, Suspected — Fig. 3 R10 0.192 Opus cell flagged (Confirmed or Suspected) Not flagged Positive values mean the relay answered fewer of the 25 selected questions correctly than the official API.
Most relays sit within a few points of the official API. Two cheap ones answer like a different model, missing more than half of what the official Opus gets right.

R20’s Opus answered 0.409 of the questions correctly against 1.000 for the official API. R11’s got 0.418 against 0.992. R01 Opus scored 0.704 against 0.992, with misses across both knowledge and capability questions. At R02, priced at the official rate, Astra moved from a gap of −0.024 in round one to +0.248 in round two, with answers saying the 2025 Nobel Prize in Literature and Super Bowl LX had not happened yet, while other trials of the same questions were correct. The authors read that as “consistent with part of the traffic reaching a model with an earlier knowledge boundary,” and they stop there.

They stop there on purpose, and they’re right to. An accuracy gap shows that the relay behaves differently. It doesn’t show which model ran. A wrapper prompt, a refusal policy, a different reasoning setting or a cache can all lower a score. R10, a same-price relay flagged in all four of the authors’ analysis variants, wraps requests in something that looks like a coding client, and its responses mention disabled tools. In an early pilot, removing one explicit refusal instruction coincided with Opus scores going from 6/40 to 20/40.

The test also has blind spots, and the authors measured them. They built simulated relays by mixing official answers with substitute answers at fixed rates and ran the same test 1,000 times per setting.

How often the accuracy test flags a simulated substituting relay Line chart from Table S2 of arXiv 2611.07650. Share of 1,000 simulated cells flagged, by nominal share of responses replaced. GPT-6 Luna replacing GPT-6 Astra: 0.0% at 10% replacement, 4.0% at 25%, 77.7% at 50%, 100% at 100%. Claude Haiku 4.5 replacing GPT-6 Astra: 47.1% at 10%, 100% at 25%, 50% and 100%. GPT-6 Luna replacing Claude Opus 5: 2.1% at 10%, 54.2% at 25%, 99.5% at 50%, 100% at 100%. The official-to-official control at 0% replacement flagged 0 of 1,000. SIMULATED CELLS FLAGGED · 1,000 PER POINT · ARXIV 2611.07650 TABLE S2 100% 50% 0% 0% 10% 25% 50% 100% Share of responses replaced by the substitute Detection rate either flag, single test Haiku 4.5 replacing Astra, 10%: 47.1% — Table S2 Luna replacing Opus, 25%: 54.2% — Table S2 Luna replacing Astra, 10%: 0.0% — Table S2 Luna replacing Astra, 25%: 4.0% — Table S2 Luna replacing Astra, 50%: 77.7% — Table S2 47.1% 54.2% 4.0% 77.7% GPT-6 Luna for Astra Claude Haiku 4.5 for Astra GPT-6 Luna for Opus Control with official answers only: 0 of 1,000 flagged. Questions were chosen on these same substitutes, so the rates are optimistic.
A weaker model from another family gets caught at 10% replacement about half the time. A cheaper sibling of the advertised model can swap in for a quarter of the traffic and almost never be noticed.

So the flags are a floor. A relay that sends one request in four for Astra to Luna passes this test 96 times out of 100, and that’s under conditions the authors describe as favorable to detection.

The permissions

Every accuracy request declared one unused tool, called noop, and set tool_choice: none. That’s the plainest instruction an API offers: do not call a tool. In round one, four relays returned tool calls anyway: 24 at R11, 4 at R17, and 1 each at R04 and R10, all on Opus. A fifth, R08, returned two web_search calls on Astra for a request that declared no tools at all. No official response in either round contained a tool call, and in round two none of the relays returned one either, including R11.

Nothing was executed, because the client never runs returned calls. The paper also can’t say where the call came from: the relay, an upstream provider ignoring the parameter, or a format translation. The authors also screened 3,332 responses to code and setup prompts for lookalike packages, odd domains, hidden shell commands and encoded payloads, and confirmed no injection in any of them.

I’d read those two results together. The screen looked at what the content said, and the permission check looked at what the response did, so a clean screen says nothing about whether the relay honored tool_choice. The paper’s recommendation follows from that: “An application should reject calls that violate its requested permissions and inspect tool names, arguments, commands, and destinations before execution.” Put another way, tool_choice: none is a request to the relay, and the client should still check what comes back. An agent framework that executes whatever tool call comes back is trusting the relay’s forwarding code with its permissions.

The bill

In round two the authors compared input-token counts for identical requests.

Input tokens reported for one identical GPT-6 Astra request Bar chart from section V-E of arXiv 2611.07650. For the same Astra question in round two, the official API reported 36 input tokens. Relays R01 and R20 each reported 4,416 input tokens and R11 reported 4,418, with 4,224 of them reported as cached in each case. REPORTED INPUT TOKENS · SAME ASTRA REQUEST · ROUND 2, 20 OCT 2026 · ARXIV 2611.07650 Official API Relay R01 Relay R20 Relay R11 36 input tokens — section V-E 4,416 input tokens — section V-E 4,416 input tokens — section V-E 4,418 input tokens — section V-E 4,224 cached — section V-E 4,224 cached — section V-E 4,224 cached — section V-E 36 4,416 4,416 4,418 Reported as cached (4,224) Reported as ordinary input R08 reports a 4,216-token cached surplus on Astra. R10 reports a median Opus surplus near 59,500, off this scale.
The question was 36 tokens long. Three relays reported a fixed 4,224 cached tokens on top of it, which is about the size of a system prompt nobody sent.

Five relays reported between about 4,200 and 60,000 more input tokens than the official API for identical requests. The authors’ reading is careful: a fixed, cached surplus of this size “is consistent with an added system prompt, but altered accounting can produce the same reports.” Either way it’s a problem. If it’s a hidden prompt, the relay is adding instructions to every request. If it’s accounting, the meter is wrong.

Next they checked whether the bill follows the meter. Six relays expose an account balance, so the authors read it before and after collection and compared the drop with the cost implied by reported usage at listed prices. All six matched to the cent. R11 is the one service in both groups, a reported surplus of more than 4,000 tokens and a readable balance, so it’s the one case where the surplus is shown to land on the bill.

R11 round-two Astra usage: share of tokens versus share of cost Paired bar chart computed from Table S3 of arXiv 2611.07650 for relay R11, GPT-6 Astra, 188 calls in round two. Ordinary input: 63,575 tokens at $1.50 per million, 7.5% of tokens and 24.1% of cost. Cache reads: 756,388 tokens at $0.15 per million, 89.5% of tokens and 28.6% of cost. Output: 25,011 tokens at $7.50 per million, 3.0% of tokens and 47.3% of cost. Total implied cost $0.3964. R11 · GPT-6 ASTRA · 188 CALLS · ROUND 2 · SHARES COMPUTED FROM ARXIV 2611.07650 TABLE S3 0% 25% 50% 75% 100% Ordinary input Cache reads Output 63,575 tok · $1.50/M 756,388 tok · $0.15/M 25,011 tok · $7.50/M 7.5% of tokens — Table S3 89.5% of tokens — Table S3 3.0% of tokens — Table S3 24.1% of cost, $0.0954 — Table S3 28.6% of cost, $0.1135 — Table S3 47.3% of cost, $0.1876 — Table S3 7.5% 89.5% 3.0% 24.1% 28.6% 47.3% Share of reported tokens Share of implied cost ($0.3964 total) R11's balance fell by exactly the cost its reported usage implies at these listed rates.
Nine in ten of the tokens R11 billed for Astra were cache reads, the category where the surplus sits. At a tenth of the input price they're a small slice of the bill, which also makes them easy to miss.

That’s why “it matched to the cent” isn’t the reassurance it sounds like. The relay reports the token count, sets the price list and runs the balance. A bill that equals reported usage times listed rates shows the relay’s arithmetic is consistent. It doesn’t show the count is true. The authors make the same point: “Agreement with the balance confirms the charge calculation, but does not validate the reported quantity.”

The money in this study is small. R11’s whole round-two Astra collection, 188 calls, cost about 40 cents. But 4,224 cached tokens on a 36-token question is a markup of more than a hundred times on input volume, and cache reads are exactly where prices have been moving. I worked through how a cache-read price cut moves different bills in an earlier post. A surplus parked in the cheapest category is also the hardest to notice on an invoice.

Cloud computing already named this

The problem is older than this market. In 2011 Vyas Sekar and Petros Maniatis argued that cloud computing needed verifiable resource accounting, meaning customers could be assured “their applications indeed physically consumed the resources they were charged for.” Chen and colleagues, in 2013, put the gap bluntly: “customers cannot verify that their invoice maps to their actual usage.” The same year Mihoob, Molina-Jimenez and Shrivastava pointed out that unlike gas or electricity, “no consumer–trusted metering services are available for cloud services, so consumers have no choice but to rely on the usage data made available by the providers.” They proposed calling an accounting model consumer-centric only if the customer could collect every input to the bill independently.

Token billing fails that test by construction. The customer can’t count what the model saw, because the relay decides what the model sees. The paper’s one external meter is the official API, used as a reference for identical requests. That works for a researcher with an official account, and it’s useless for the user who picked a relay because they can’t get one.

What I’d do with this

If I were routing production traffic through a relay, I’d take three things from the paper.

First, I’d send a fixed, short request through the relay and the official API every so often and compare the reported input tokens. A surplus of 4,224 on a 36-token prompt doesn’t need a statistician to spot. Second, I’d enforce tool permissions in my own client: if I sent tool_choice: none, any returned tool call is an error, not an action. Third, I’d treat a relay’s accuracy as something that changes. Of the 35 relay-model cells measured in both rounds, 9 were flagged both times, 5 lost their flag and 2 gained one. R08’s and R17’s Astra gaps vanished between rounds while their Opus cells stayed flagged. One clean test eleven days ago says little about today.

The limits are the authors’ own. The 20 relays aren’t a random sample. Requests came from one country, were standalone and benign, and never executed a tool, so attacks that need credentials, long conversations or real tool use wouldn’t show up. The services are anonymized. And no measurement here identifies a backend model. The paper doesn’t claim any relay committed fraud, and I don’t either.

What it does show is narrower and more useful. The model label, the price list and the invoice all come from the relay, so their agreeing with each other proves little. Every problem the paper found came from comparing the relay against the official API, which the relay doesn’t control.

References

  1. Wu, X., Zhao, Z., Li, Q., Cheong, C., Li, X. and Ni, J. (2026). What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays. arXiv:2611.07650, submitted 5 November 2026.
  2. Sekar, V. and Maniatis, P. (2011). Verifiable resource accounting for cloud computing services. Proceedings of the 3rd ACM Workshop on Cloud Computing Security: 21–26, published 21 October 2011.
  3. Chen, C., Maniatis, P., Perrig, A., Vasudevan, A. and Sekar, V. (2013). Towards verifiable resource accounting for outsourced computation. Proceedings of the 9th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments: 167–178, published 16 March 2013.
  4. Mihoob, A., Molina-Jimenez, C. and Shrivastava, S. (2013). Consumer-centric resource accounting in the cloud. Journal of Internet Services and Applications 4: 8, published 11 March 2013.