A paper by Dario Caldara of the Federal Reserve Board and Jesús Fernández-Villaverde of Penn, dated 16 November and titled Can Peer Review Survive AI-Generated Text?, was being argued over on Hacker News today. Its argument is that economics journals should make AI detectors a standard tool, and that a referee report written by a language model should be handled like plagiarism. Both are journal editors. They write that they have already handled papers and reports that the detector Pangram scored as 99 percent AI-generated, and that as authors they have received mostly AI-generated referee reports, “even from the top journals in the profession.”
I agree with most of the policy. The part I want to look at is the evidence they give for trusting the detector, because the paper runs together two claims of very different strength. One is that the detector catches reports a model wrote. The other is that it doesn’t flag reports a person wrote. The experiment supports the first well. On the second it can say much less than it seems to, and a journal that wants to reject reports automatically depends on the second.
The experiment
The authors drew 25 first-round referee reports from the Corpus of Transparent Peer Review, a CC BY 4.0 dataset of the reports Nature Communications published for 560 papers in the biological and physical sciences. All the reports were written between 2018 and mid-2020, before ChatGPT existed, so they are known to be human. They run from 256 to 1,310 words.
Research assistants put each report through four prompts, once with Claude Sonnet 5.5 and once with ChatGPT 5.6 Terra, each run in a fresh chat with memory turned off. Prompt 1 asked for proofreading only. Prompt 2 asked for language editing “for clarity and flow” with every point kept. Prompt 3 asked the model to rewrite the report in its own words. Prompt 4 gave the model only the published paper and asked for a referee report “with three main points and a few minor points.” That makes 50 versions per prompt, all scored with Pangram 4.1 in November 2026.
The gradient is clean. Proofreading changed a median 4 percent of words and left every version scored fully human. Language editing changed about 30 percent of the words, and the reports still averaged 95 percent human, with five showing a small AI-generated share. Rewriting gave mixed results, and here the two models differed: Claude’s rewrites averaged 40 percent AI-generated, ChatGPT’s 66 percent. Reports written from the paper alone came out 98 percent AI-generated on average, and all 50 were classified as mostly AI.
So if a referee hands the whole job to a model, this detector catches it. With these two models, on reports of this length, the catch rate was 50 out of 50.
The result that needs no detector
The finding I’d put first sits in a paragraph that opens with “Strikingly.” Prompt 4 never asked for a recommendation. All 50 generated reports gave one anyway, and all 50 said major revision, for papers that had already been through review and been published.
Human referees disagree with each other all the time, and editors rely on that. The authors’ complaint is “the lack of variation in judgment,” and I think it’s their strongest point because it doesn’t depend on any classifier. An editor who sends a paper to three referees is buying three independent readings. If two of those readings come from the same model with roughly the same prompt, the editor gets one opinion counted twice, and in this sample that opinion was major revision every time. The authors add that the cost may fall hardest on innovative work, since a paper that departs from the literature gives a model more deviations from standard practice to list. Their sample can’t test that, but it fits what they saw.
What zero out of twenty-five can tell you
The second claim is about false positives. The authors write that a long text flagged as AI-generated “is almost certainly generated by AI,” that false positives “come only from errors of the detector, which are rare,” and that they “have not encountered a single false positive.” In the experiment, all 25 originals were scored 100 percent human.
That’s a good result. It’s also a small denominator. If a detector produced no false positives in 25 independent human texts, the most you can say at the usual 95 percent level is that its false-positive rate is below about 11.3 percent. The bound comes from solving (1 − p)25 = 0.05. To push the bound down to the 0.5 percent level that matters for automatic decisions, you’d need about 600 clean texts with no flags at all.
The authors don’t rest the low error rate on their 25 reports alone. They cite Brian Jabarian and Alex Imas, whose NBER working paper (September 2025) tested Pangram, Originality.ai, GPTZero and an open-source RoBERTa model on a large corpus across genres, lengths and models. Its abstract reports that Pangram had “near-zero FNR and FPR rates” and that it was “the only tool to satisfy a strict cap (FPR ≤ 0.005) without sacrificing accuracy.” That is real evidence for this detector. But their corpus wasn’t referee reports, and the experiment in the new paper, which looks like it is testing that question directly, doesn’t have the sample to do it.
False-positive rates belong to populations
A false-positive rate also isn’t a fixed property of a detector. It depends on the detector and on the kind of writing it’s given, and the literature has cases where the rate swings widely.
The best-known one is by Weixin Liang, James Zou and coauthors, who wrote in Patterns in 2023. They ran seven widely used GPT detectors on 91 TOEFL essays by non-native English writers and 88 US eighth-grade essays. The detectors were close to perfect on the American essays. On the TOEFL essays they returned an “average false-positive rate: 61.3%.” At least one detector flagged 97.8 percent of the TOEFL essays, and all seven together flagged 19.8 percent. The mechanism they found was perplexity: writers with a smaller working vocabulary produce more predictable text, and predictable text is what these detectors read as machine output. When the authors had ChatGPT enrich the vocabulary of the same essays, the average false-positive rate fell to 11.6 percent.
A study in Computers & Education this year by Yicheng Sun, Yihan Liao and Xiaoxue Ma tested 13 detectors on more than 280,000 pieces of real student work. Their results agree with Caldara and Fernández-Villaverde in one way: under adversarial editing, most tools kept “a consistently low and acceptable false positive rate, typically ranging from 0% to 4%” on human text, even as up to 88 percent of edited AI text got past them. They also report false-positive rates above 40 percent for commercial tools on short coursework, and bias against STEM writing because of how formulaic it is.
The errors also tend to run one way. Debora Weber-Wulff and colleagues tested 14 detection tools for the International Journal for Educational Integrity in 2023 and found them biased toward calling text human rather than catching AI text, and obfuscation made that worse. That fits the authors’ point that misses are more common than false alarms.
Polishing is a third case. Jinge Wang and coauthors, in a 2025 letter in JAAD International, took dermatology research letters and asked ChatGPT to “enhance the readability and flow” while keeping their meaning, which is close to the new paper’s prompt 2. GPTZero classified 97 to 100 percent of the originals as human, and then flagged 75 to 85 percent of the polished versions as AI-generated. In the referee-report experiment, Pangram scored the equivalent edit about 1 percent AI.
None of this shows that Pangram misfires on referee reports. The older studies mostly test other detectors, and Jabarian and Imas found that Pangram held up on the kinds of text that broke the others. What it shows is that “false positives are rare” is a claim about one detector on one population of texts, and that the populations where detectors have failed look a lot like economics refereeing. The writing is technical and formulaic, and many referees write in a second language. The authors count themselves among those for whom English isn’t a first language, and say AI editing helps them. The 25 Nature Communications reports tell us nothing about who wrote them.
Why the bound matters once flags have consequences
How much a flag is worth depends on the base rate as well as the error rate. Panagiotis Tsigaris and Jaime Teixeira da Silva make this argument in Next Research this year, borrowing the screening arithmetic from epidemiology. If only a minority of the texts a detector sees contain detectable, undeclared AI writing, then even a small false-positive rate fills the flagged pile with innocent texts. They work with detector-facing prevalence of 10 to 20 percent.
To see how much the false-positive rate matters here, take the low end of that range: suppose 10 percent of referee reports a journal receives are fully delegated, and give the detector a perfect catch rate, as in the experiment. The share of flagged reports that really are AI-written then depends entirely on how often human reports get flagged.
If Pangram really is at the 0.5 percent level on referee reports, a flag is strong evidence, and the authors’ reasoning holds. But 0 out of 25 is also consistent with a rate where half the flagged reports are innocent. For papers, the authors handle this well. They say a flag “should trigger a closer reading, not a rejection,” and that sanctions should follow the errors that scrutiny finds rather than the flag. A false positive then costs some editor time.
For referee reports, they go further. One of their proposals is that “a journal’s submission system could automatically reject a referee report whose AI-generated share is above a small threshold, such as 10 percent.” Elsewhere in the same paper, their rule of thumb for editors is to ignore shares below 20 percent and read carefully between 20 and 50. Their own table makes a 10 percent line look tight: language editing, which they want journals to publish as an example of acceptable use, already produced a small AI-generated share in five of 50 reports. The paper reports averages, so I can’t tell how close any one of those came to 10 percent. An automatic cutoff is where the false-positive rate stops being an abstraction, and it’s the place where the paper has the least evidence to set one.
What I’d take from it
The policy itself is well argued. Putting the burden of verification on the author, treating a flag as a reason to read closely rather than a verdict, and telling referees that a delegated report “adds absolutely nothing” all seem right to me. The unanimous “major revision” is the clearest evidence in the paper for why AI-written reviews damage the process, and that argument needs no detector at all.
What I’d change is how the detector gets introduced. A journal that wants to rely on flags should first measure the false-positive rate on its own archive: a few hundred referee reports written before late 2022, in its own field, from its own mix of native and non-native writers. Around 600 clean reports with no flags would support the 0.5 percent level. The Corpus of Transparent Peer Review the authors used covers 560 papers, so even one first-round report per paper would get close to that bound, and the authors have already done the matching. It’s a small amount of work next to the cost of telling an honest referee their report was rejected by software.
References
- Caldara, D. and Fernández-Villaverde, J. (2026). Can Peer Review Survive AI-Generated Text?. Working paper, 16 November 2026.
- Akbas, E., Sönmez, D. and Tekin, M. (2024). Corpus of Transparent Peer Review. Mendeley Data, version 1, CC BY 4.0.
- Jabarian, B. and Imas, A. (2025). Artificial Writing and Automated Detection. NBER Working Paper 34223, September 2025.
- Liang, W., Yuksekgonul, M., Mao, Y., Wu, E. and Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns, 4, 100779, July 2023.
- Sun, Y., Liao, Y. and Ma, X. (2026). Trusting AI to detect AI? A systematic evaluation of the reliability and robustness of current AIGC detection tools for student academic work. Computers & Education, 249, 105616, August 2026.
- Wang, J., Jairath, N., Shue, E., Perrotta, P. L. and Hu, G. (2025). ChatGPT-polished writing boosts the risk of human-authored manuscripts being miscredited as AI-generated. JAAD International, 22, 23-25, October 2025.
- Tsigaris, P. and Teixeira da Silva, J. A. (2026). AI detecting AI in academic writing: Why most AI detector findings are false. Next Research, 7, 101396, May 2026.
- Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P. and Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19, 26, December 2023.