Sixty of Sixty-Six Cost Ranges Overlap. The JavaScript Bytes Barely Do.

On 28 October Nolan Lawson published “The AI cost of JavaScript frameworks,” a benchmark written against the vibes. His complaint is that frameworks get called “better for agents” without evidence: “Did anyone think to ask the agents?” So he asked them. He wrote a spec for a small Pokémon browser backed by a local snapshot of the PokéAPI, gave it to Opus 5.5 at high effort along with a conformance test suite, and had the agent build the app in twelve frameworks: Angular, Astro, Gracile with Lit, Hono with htmx, Next.js, Nuxt, Qwik, React Router, SolidStart, SvelteKit, TanStack Start, and “vanilla” TypeScript on Vite. Each framework was built five times, and he reports the median.

The headline ordering is easy to quote. Hono+htmx was cheapest at $2.67 a run, Angular the most expensive at $4.97. Hono+htmx also wrote the least code, and Next.js shipped the most JavaScript. What made me want to write about it is that he also published the range for every number, the cheapest and dearest of the five runs, and admitted where it hurt: “everything I’m measuring is very stochastic.” Most of the medians don’t survive a look at those ranges, and the one that does is the most useful number in the post.

What the setup holds fixed

The harness is careful in the places that matter. The prompt that each agent got differs only in the framework’s name. The pinned benchmark source reads: “Implement the app described in SPEC.md using ${label}. Use the latest stable version. Work autonomously until npm test passes.” The conformance suite checks the accessibility tree rather than the DOM, so a framework built on shadow DOM isn’t penalised for structure. Each agent ran in a sandbox, didn’t know it would be scored in Lighthouse, and was told not to pre-render every page as static HTML, which Lawson notes would otherwise beat the whole benchmark.

He also installed each framework’s official MCP server or agent skills where one existed, so no framework could complain it was underrepresented. Whether the agent used them is another matter. In his table the Nuxt MCP was never used, Next.js agents read the skill file in one run of five, and TanStack Start’s agent found its bundled skill files in two runs of five. React Router and Hono’s skills were used in every run, and the Svelte MCP in every run.

Angular is the exception in the sample. In two runs the agent built a server-rendered app that refused non-localhost requests, though the spec says “Assume a reverse proxy will sit in front of your server in production.” Lawson counted those as spec failures and did not re-run them, so Angular’s numbers are medians of the three runs that worked.

The cost ranges mostly overlap

Here is his cost table drawn with its ranges, sorted by median.

Cost per build run, median and range, for twelve frameworks Dot and range plot on a linear dollar scale from 2 to 7 dollars. Medians with min to max ranges: Hono+htmx 2.67 (2.23 to 3.19), Astro 2.76 (2.16 to 3.63), Nuxt 2.82 (2.39 to 4.25), Vanilla 2.98 (2.24 to 3.37), Next.js 3.14 (2.58 to 4.13), Qwik 3.27 (2.35 to 3.80), TanStack Start 3.54 (3.11 to 3.96), SolidStart 3.79 (2.84 to 4.81), React Router 3.81 (2.82 to 3.97), SvelteKit 4.28 (3.59 to 4.65), Gracile+Lit 4.65 (3.92 to 5.07), Angular 4.97 (3.19 to 6.82). Most ranges overlap; only SvelteKit and Gracile+Lit sit entirely above the cheapest framework's range. COST PER RUN, USD · MEDIAN AND MIN–MAX · LAWSON, 28 OCT 2026 $2 $3 $4 $5 $6 $7 Hono+htmx Astro Nuxt Vanilla Next.js Qwik TanStack Start SolidStart React Router SvelteKit Gracile+Lit Angular (3 runs) Hono+htmx: median $2.67, range $2.23 to $3.19 — Lawson, 28 Oct 2026 Astro: median $2.76, range $2.16 to $3.63 — Lawson, 28 Oct 2026 Nuxt: median $2.82, range $2.39 to $4.25 — Lawson, 28 Oct 2026 Vanilla: median $2.98, range $2.24 to $3.37 — Lawson, 28 Oct 2026 Next.js: median $3.14, range $2.58 to $4.13 — Lawson, 28 Oct 2026 Qwik: median $3.27, range $2.35 to $3.80 — Lawson, 28 Oct 2026 TanStack Start: median $3.54, range $3.11 to $3.96 — Lawson, 28 Oct 2026 SolidStart: median $3.79, range $2.84 to $4.81 — Lawson, 28 Oct 2026 React Router: median $3.81, range $2.82 to $3.97 — Lawson, 28 Oct 2026 SvelteKit: median $4.28, range $3.59 to $4.65 — Lawson, 28 Oct 2026 Gracile+Lit: median $4.65, range $3.92 to $5.07 — Lawson, 28 Oct 2026 Angular: median $4.97, range $3.19 to $6.82 — Lawson, 28 Oct 2026 $2.67 $2.76 $2.82 $2.98 $3.14 $3.27 $3.54 $3.79 $3.81 $4.28 $4.65 $4.97 Dashed line: the most expensive Hono+htmx run ($3.19). API list prices as reported by Claude Code, not money spent.
Read the bars, not just the dots. Only SvelteKit and Gracile+Lit sit wholly to the right of the dashed line.

I counted the overlaps. With twelve frameworks there are 66 pairs. In 60 of them the two cost ranges overlap or meet. The cheapest framework’s range, Hono+htmx at $2.23 to $3.19, reaches into the ranges of nine of the other eleven. Only SvelteKit, whose cheapest run cost $3.59, and Gracile+Lit, at $3.92, never came in under Hono+htmx’s dearest run. Angular’s cheapest surviving run cost exactly $3.19.

Lawson says as much in passing: React Router’s median is about a dollar worse than Hono+htmx’s, but its cheapest run ($2.82) beat Hono+htmx’s dearest ($3.19). His firmer conclusion is that “SvelteKit, Gracile+Lit, and Angular are more expensive to build than the other frameworks.” The ranges back that for SvelteKit and Gracile+Lit. For Angular they’re weaker: its range starts where Hono+htmx’s ends, and it’s a median of three runs that excludes the two where the agent failed the spec.

Time and tokens look the same. Time ranges overlap in 62 of 66 pairs. Token ranges overlap in 60, and the framework with the lowest median token count, Nuxt at 3.85 million, has a range (3.16 to 8.18 million) that touches every other framework’s.

Two caveats. Min-to-max over five runs is a wide interval by construction, and overlapping ranges don’t prove two medians are equal. They do show how little five runs can tell apart. Second, Lawson’s explanations for the expensive ones are concrete and plausible: SvelteKit agents read the SvelteKit 3 migration guide in every run, he thinks because the release is newer than the model’s training data, and Gracile agents cloned the Gracile repository or read its compiled source from node_modules. Those are framework effects, but what they measure is how well the model knows each framework, and that changes with every model release.

The bytes don’t overlap

Here is the same plot for the JavaScript each finished app sent to the browser, as measured by Lighthouse.

JavaScript transferred per app, median and range, for twelve frameworks Dot and range plot on a linear KiB scale from 0 to 150. Medians with min to max ranges: Astro 0 (0 to 1), Vanilla 7 (6 to 7), Gracile+Lit 8 (1 to 13), Hono+htmx 18 (18 to 18), SvelteKit 36 (35 to 36), SolidStart 40 (38 to 40), Qwik 40 (40 to 41), Nuxt 84 (83 to 85), Angular 93 (81 to 102), React Router 115 (115 to 119), TanStack Start 118 (112 to 119), Next.js 142 (136 to 143). Most ranges are a few KiB wide and do not overlap. JS TRANSFERRED, KIB · MEDIAN AND MIN–MAX · LAWSON, 28 OCT 2026 0 50 100 150 Astro Vanilla Gracile+Lit Hono+htmx SvelteKit SolidStart Qwik Nuxt Angular (3 runs) React Router TanStack Start Next.js Astro: median 0 KiB, range 0 to 1 KiB — Lawson, 28 Oct 2026 Vanilla: median 7 KiB, range 6 to 7 KiB — Lawson, 28 Oct 2026 Gracile+Lit: median 8 KiB, range 1 to 13 KiB — Lawson, 28 Oct 2026 Hono+htmx: median 18 KiB, range 18 to 18 KiB — Lawson, 28 Oct 2026 SvelteKit: median 36 KiB, range 35 to 36 KiB — Lawson, 28 Oct 2026 SolidStart: median 40 KiB, range 38 to 40 KiB — Lawson, 28 Oct 2026 Qwik: median 40 KiB, range 40 to 41 KiB — Lawson, 28 Oct 2026 Nuxt: median 84 KiB, range 83 to 85 KiB — Lawson, 28 Oct 2026 Angular: median 93 KiB, range 81 to 102 KiB — Lawson, 28 Oct 2026 React Router: median 115 KiB, range 115 to 119 KiB — Lawson, 28 Oct 2026 TanStack Start: median 118 KiB, range 112 to 119 KiB — Lawson, 28 Oct 2026 Next.js: median 142 KiB, range 136 to 143 KiB — Lawson, 28 Oct 2026 0 KiB 7 KiB 8 KiB 18 KiB 36 KiB 40 KiB 40 KiB 84 KiB 93 KiB 115 KiB 118 KiB 142 KiB Home page only, Lighthouse mobile run. Gracile+Lit's spread comes from two runs where the agent didn't use Lit.
Five builds of the same framework ship nearly the same bytes. The ordering here would survive a rerun.

Only 5 of the 66 pairs overlap here. Most frameworks’ five builds land within a few KiB of each other: Hono+htmx shipped 18 KiB every time, Next.js between 136 and 143. The wide ones have reasons. Angular’s range runs from 81 to 102 KiB, and Lawson notes that the Gracile agent didn’t use Lit at all in two runs, which explains its spread from 1 to 13 KiB.

Lines of code fall in between, with 42 of 66 pairs overlapping, which fits: the framework sets a floor on boilerplate, and the agent decides the rest. Lawson’s “~70%” comparison holds up on the medians, since Hono+htmx’s 1,370 lines are 68 percent of Angular’s 2,009, and the two ranges don’t come close.

How many of the 66 framework pairs have overlapping ranges, by metric Horizontal bars, out of 66 pairs of frameworks, counting pairs whose min to max ranges overlap or meet. Wall-clock time: 62. Cost: 60. Tokens: 60. Lines of code: 42. JavaScript bytes transferred: 5. Computed from the medians and ranges in Lawson's 28 October 2026 tables. PAIRS OF FRAMEWORKS WHOSE RANGES OVERLAP · OUT OF 66 · FROM LAWSON'S TABLES Wall-clock time Cost Tokens Lines of code JavaScript bytes 62 of 66 pairs overlap — computed from Lawson, 28 Oct 2026 60 of 66 pairs overlap — computed from Lawson, 28 Oct 2026 60 of 66 pairs overlap — computed from Lawson, 28 Oct 2026 42 of 66 pairs overlap — computed from Lawson, 28 Oct 2026 5 of 66 pairs overlap — computed from Lawson, 28 Oct 2026 62 60 60 42 5 A pair counts if one framework's min–max range overlaps or meets the other's. Light line marks all 66.
The metrics the agent controls barely separate the frameworks. The metric the framework controls separates almost all of them.

What the split means

I think the explanation is simple. The JavaScript a page ships is mostly decided by the framework’s runtime and whether it renders on the server or the client, and the agent can’t do much to change it. Cost, time and tokens depend on how the agent’s session went: whether it went “off on some weird tangent,” in Lawson’s words, whether it read the docs, whether it happened to find the skill file. All of that is agent behavior, and it varies a lot from run to run.

The literature has measured this for single prompts. Ouyang, Zhang, Harman and Wang ran 829 code generation problems through ChatGPT repeatedly and found that for 75.76 percent of CodeContests tasks, no two requests produced equal test output under default settings, and that “setting the temperature to 0 does not guarantee determinism in code generation.” Donato and colleagues, testing 548 Java methods, found sampling settings changed results significantly and offered guidance on how many times a prompt has to be repeated to account for the non-determinism. A ten-minute agent session that compounds thousands of such samples will vary more than any single completion does, which is what Lawson’s ranges show.

This is the same trap I wrote about in the skills study, where three runs per task were enough to make noise look like a finding. Lawson avoided most of it by publishing his ranges and saying plainly where they’re wide. The benchmark is public on GitHub and he invites reruns.

What I’d take from it

If you’re picking a framework for agent-written apps, I’d split the evidence in two. The quality numbers, meaning the JavaScript bytes, Total Blocking Time and LCP, depend on the framework and will repeat. On this app the gap between Astro’s 0 KiB and Next.js’s 142 KiB will still be there on the next model. The cost numbers are a property of a particular model, at a particular effort level, meeting particular framework docs. Five runs separate the two expensive outliers and little else. A SvelteKit release that’s newer than the model’s training data costs extra until the training data catches up.

Lawson’s own suspicion, that “the current crop of MCPs and skills may be doing more harm than good,” is the same kind of question. The usage table shows agents ignoring the tools as often as using them, so the benchmark as run can’t answer it. A version that turned each framework’s tooling on and off, at ten runs per cell rather than five, could.

His larger point stands. He asked for evidence over vibes and then published enough of it that a reader can check his medians, and the checked version is narrower than the headline. Framework choice clearly decides what your users download. Whether it decides what your agent spends is still unclear.

References

  1. Lawson, N. (2026). The AI cost of JavaScript frameworks. nolanlawson.com, 28 October 2026.
  2. Lawson, N. (2026). agent-framework-benchmark: bench/frameworks.mjs and template/SPEC.md. GitHub, pinned commits accessed 1 November 2026.
  3. Ouyang, S., Zhang, J. M., Harman, M. and Wang, M. (2025). An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Transactions on Software Engineering and Methodology, published January 2025.
  4. Donato, B., Mariani, L., Micucci, D. and Riganelli, O. (2025). Studying How Configurations Impact Code Generation in LLMs: The Case of ChatGPT. 2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC), April 2025.