Evan X. Merz’s post “AI usage generates a fragile codebase,” dated 7 November, was near the top of Hacker News today. It starts with a sentence I keep rereading. Writing about his last months at Meta, before he was laid off in May, he says: “I generated at least 90% of my code using AI in my last few months there, which was the minimum for a decent rating.” At his current employer, also a large tech company, he has “fully made the switch to generating 100% of my code using AI because that’s the expectation today.”
The rest of his post makes eight claims about AI-heavy codebases: four causes and four symptoms. AI writes shallow tests, it over-abstracts, people stop understanding the code, and review can’t keep up. The symptoms are more incidents, slower recovery, errors that only users see, and worse products. His central mechanism is one sentence: “AI tends to solve the problems that can fit within its context window very well, but it’s totally oblivious to the long term issues that can’t be understood in that relatively small frame.”
It’s one engineer’s account, and he says himself that he can “only speculate” about why he was laid off. I wanted to know how well the claims hold up against studies that measured something. The answer depends on how large a unit each study measured, and that turns out to be his argument too.
Small units look fine
The studies with the cleanest designs measure the smallest units. An ESEM 2025 paper by Molison and colleagues ran Python solutions from LLMs and from humans through SonarQube, across introductory, interview and competition problems. The headline result: “LLM-generated code has fewer bugs and requires less effort to fix them overall.” The exception was at the hardest level: “In competition-level problems, the LLM solutions sometimes introduce structural issues that are not present in human-written code.” An ICSE 2025 study of six models on HumanEval found that their errors were often “non-trivial, multi-line” and had “various root causes.” HumanEval problems are single functions.
That is roughly what Merz predicts. Inside one file and one prompt, the code is as good as a human’s or better. The trouble starts where the problem stops fitting in the frame.
One level up are the randomized trials, which time a developer on a task. Google ran one with 96 of its full-time engineers on an enterprise-grade task, and AI cut the time spent by “about 21%.” The authors add that “our confidence interval is large” and warn that a result from internal tooling in summer 2024 may not carry over. METR’s July 2025 trial with 16 experienced open-source developers in mature repositories went the other way: allowing AI “increases completion time by 19%,” while the developers themselves estimated afterwards that it had saved them 20%. I went through the METR trials and their February 2026 follow-up in an earlier post.
The trials can’t settle Merz’s claims, and nobody designed them to. They stop when the task is done. What he describes starts after that, when the change is merged, other changes are built on it, and someone has to debug it at 2 a.m.
The one measurement at team scale
The largest dataset that looks past the task is Google’s 2024 DORA report, which surveys software teams every year. It found that a 25% increase in AI adoption was associated with a 7.5% increase in documentation quality, 3.4% in code quality and 3.1% in code review speed. The same increase came with “an estimated decrease in delivery throughput by 1.5%, and an estimated reduction in delivery stability by 7.2%.”
That chart puts numbers on Merz’s argument. Code quality, documentation and review speed all describe the work as the developer sees it, and they go up. Stability describes what happens after deployment, across many changes and many people, and it goes down. DORA’s own explanation is close to his: improving the development process “does not automatically improve software delivery,” at least not without “small batch sizes and robust testing mechanisms.” His fourth cause, review falling behind the pace of generation, is a batch-size problem.
I’d hold this loosely. It’s a survey. “Associated with” is the report’s phrase and the right one, because teams that adopt AI faster may differ in ways that also make them less stable. The tools it describes are two years old, from before agents wrote whole pull requests. The same report says 39% of respondents had little or no trust in AI-generated code. Even so, it is the one team-scale number I found, and it points the way Merz says.
The claim nobody has measured
Merz’s strongest sentence is “AI coding agents are the reason why there are more incidents impacting customers.” In his telling it was one P0 at the end of a month, then two SEVs the next, until “it seems like there’s an incident every weekend.” None of the sources I read can confirm or refute that. A trial that ends when the task is done can’t see an incident six months later. A survey can link adoption to stability but can’t separate the tool from the teams that rushed to adopt it. The companies that could answer it, by comparing incident rates and recovery times across services with different shares of AI-written code, are the companies setting AI-share targets. As far as I know, none of them has published that comparison.
That is why the 90% sentence bothers me more than the incident claim. A target of 90% AI-generated code counts the input. It rewards the part of the work the studies say is fine, the code inside the frame, and says nothing about the part that DORA’s stability numbers and Merz’s incident log are about. Once the share is part of a performance rating, the share goes up whatever happens to the codebase, because that’s how a target works. His own example is a small one. One of the best programmers on his team tried to merge code “that used javascript events for mouseover CSS effects rather than using the :hover selector.”
That example got an answer the same day from a group with no stake in his argument. Google Chrome’s Modern Web Guidance repository, which also made the Hacker News front page today, exists because “coding agents often default to older patterns because LLM training data contains vast amounts of legacy code,” and that leads them to “generate bloated JavaScript for tasks that now have native, high-performance web platform solutions.” It describes itself as stewarded by the Chrome, Edge, Firefox and MDN teams. Browser vendors are now shipping skills to push agents back toward the platform. The problem Merz saw on his team is common enough that the people who build browsers are writing fixes for it.
What I’d measure instead
If I ran an engineering organization with an AI-share target, I would keep the target only if it sat next to the numbers it might damage: change failure rate, time to restore service, and the share of incidents traced to changes nobody on call had read. DORA has tracked those for a decade. Track them by service, split by how much of each service’s recent code was generated, and publish the result internally. If Merz is wrong, that table will show it within a couple of quarters. If he’s right, it will show that too, and it’s better to see it in the table than in the on-call schedule.
The studies so far measured the parts of software that fit in a prompt, and there the results run from mixed to good. The fragility he describes would show up in everything that doesn’t fit, and that part hasn’t been measured yet.
References
- Merz, E. X. (2026). AI usage generates a fragile codebase. evanxmerz.com, 7 November 2026.
- Google Cloud (2024). Announcing the 2024 DORA report. Google Cloud Blog, 22 October 2024.
- Paradis, E. et al. (2025). How Much Does AI Impact Development Speed? An Enterprise-Based Randomized Controlled Trial. ICSE-SEIP 2025, April 2025.
- Becker, J. et al. (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR, arXiv:2507.09089, 12 July 2025.
- Molison, A. S. et al. (2025). Is LLM-Generated Code More Maintainable & Reliable Than Human-Written Code?. ESEM 2025, October 2025.
- Wang, Z. et al. (2025). Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models. ICSE 2025, April 2025.
- Google Chrome (2026). Modern Web Guidance. GitHub repository, accessed 10 November 2026.