The Deskilling Evidence Is Six Points in Three Months

An essay from the SigNoz newsletter, “AI Isn’t Replacing SREs. It’s Deskilling Them.”, was widely discussed on Hacker News on 30 October. It was first published on 28 February. The question it asks is a fair one: “When AI handles 95% of your incident response, do you get worse at handling the 5% that actually matters?” It answers yes, starting from Lisanne Bainbridge’s 1983 paper “Ironies of Automation” and then pointing to evidence from two fields, medicine and aviation.

I think the conclusion is probably right. I wanted to see how much the cited evidence supports it, because a claim like this gets repeated in planning documents and hiring debates long after anyone has checked the footnotes. So I opened the sources the essay links and a few it doesn’t.

The colonoscopy number holds, but it is a short window

The essay says that “endoscopists who used AI assistance for polyp detection saw their unassisted detection rates drop from 28% to 22% after a period of AI use.” The link goes to Budzyń and colleagues in The Lancet Gastroenterology & Hepatology, published online on 12 August 2025. The figures are correct. The adenoma detection rate of colonoscopies done without AI fell from 28.4% (226 of 795) to 22.4% (145 of 648).

This was a retrospective, observational study at four endoscopy centres in Poland that were part of the ACCEPT trial. The centres introduced AI polyp detection at the end of 2021. The authors compared non-AI colonoscopies from the three months before introduction with non-AI colonoscopies from the three months after. So “a period of AI use” means three months. The absolute drop was 6.0 percentage points, with a 95% confidence interval from 1.6 to 10.5 points. In the regression, exposure to AI had an odds ratio of 0.69. Patient age of 60 or more had an odds ratio of 3.60, and male sex 1.78, so patient age and sex were stronger predictors of the detection rate than AI exposure was.

Adenoma detection rate without AI, before and after AI was introduced Two bars from Budzyń and colleagues, 2025. Non-AI colonoscopies in the three months before AI introduction had an adenoma detection rate of 28.4 percent, 226 of 795. Non-AI colonoscopies in the three months after had 22.4 percent, 145 of 648. Below, the difference of minus 6.0 percentage points is drawn with its 95 percent confidence interval, which runs from minus 10.5 to minus 1.6 points. ADENOMA DETECTION RATE · NON-AI COLONOSCOPIES · 4 CENTRES, POLAND · BUDZYŃ ET AL. 2025 3 months before AI 226 of 795 3 months after AI 145 of 648 28.4% (226 of 795) — Budzyń et al., Lancet Gastroenterol Hepatol, 2025 22.4% (145 of 648) — Budzyń et al., Lancet Gastroenterol Hepatol, 2025 28.4% 22.4% Difference, percentage points, with 95% confidence interval −6.0 points (95% CI −10.5 to −1.6), p=0.0089 — Budzyń et al., 2025 −12 −9 −6 −3 0 −10.5 −6.0 −1.6 Retrospective and observational: different patients in each window, no randomisation of the comparison.
The drop is real and statistically significant, but the interval runs from under 2 points to over 10, and it was measured over one three-month window.

The authors were careful in their own wording: “Continuous exposure to AI might reduce the ADR of standard non-AI assisted colonoscopy, suggesting a negative effect on endoscopist behaviour.” The two words doing the work there are might and behaviour. A three-month dip can be skill loss. It can also be endoscopists going less carefully through the cases they know the machine isn’t watching, which is a habit and can be reversed. The study can’t separate those, and it doesn’t claim to. The journal printed a correction to the paper in November 2025, and three letters about it, with an authors’ reply, in December. I’d read the essay’s line as “one observational study found a six-point drop over three months,” which is weaker than “they got worse at the thing they were supposed to be experts in.”

The essay’s aviation sentence reads: “long-haul pilots who rely heavily on autopilot systems experience measurable degradation in situational awareness and manual flying ability. The problem got serious enough that the FAA now mandates more manual flying time to counteract the effect.”

The link goes to a slide deck from the Flight Safety Foundation’s 2019 International Air Safety Summit, by Julia Behrend and Camille Lafargue. It is an eye-tracking study in a full-flight simulator with 20 A340 pilots. Each flew 19 approaches with the flight director on and 19 with it off, and autopilot and autothrust off in both cases. The conclusions on the slides are that “The flight directors improved the precision of the approaches,” reduced sidestick inputs, and shifted where pilots looked. That measures what a guidance aid does while it is switched on. It doesn’t follow pilots over time to see whether heavy autopilot use wears down their skill. The deck mentions that claim only in passing, citing other work.

The FAA sentence has no link. The FAA’s safety alert on this subject, SAFO 13002 of 4 January 2013, is titled “Manual Flight Operations.” Its purpose line says: “This SAFO encourages operators to promote manual flight operations when appropriate.” The header says a SAFO “may include recommended action.” The alert’s concern matches the essay’s: “continuous use of autoflight systems could lead to degradation of the pilot’s ability to quickly recover the aircraft from an undesired state.” But it is a recommendation to airlines, not a mandate for more manual flying time.

What the essay claims against what each source shows A three-row ledger. Endoscopy: the essay says detection fell from 28 to 22 percent after a period of AI use; the source, Budzyń and colleagues 2025, is a retrospective study with three months before and after and 1,443 non-AI colonoscopies, and the figures match. Aviation: the essay says long-haul pilots relying on autopilot show measurable degradation; the source is a 2019 summit slide deck on 20 A340 pilots in a simulator comparing flight director on and off, which does not measure decay over time. FAA: the essay says the FAA now mandates more manual flying time; SAFO 13002 of January 2013 encourages operators to promote manual flight operations. SIGNOZ ESSAY, 28 FEB 2026 · CLAIMS CHECKED AGAINST CITED OR NAMED SOURCES Field Essay says Source shows Endoscopy Aviation FAA Unassisted detection fell 28% → 22% "after a period of AI use" Autopilot reliance causes "measurable degradation" in long-haul pilots "now mandates more manual flying time" Figures match: 28.4% → 22.4% Retrospective; 3 months before vs after; 1,443 non-AI colonoscopies 20 A340 pilots, flight director on vs off 2019 summit slides; measures the aid while on, not decay over time SAFO 13002 (Jan 2013) "encourages operators to promote manual flight operations when appropriate" Sources: Budzyń et al. 2025; Behrend & Lafargue, IASS 2019; FAA SAFO 13002.
One of the three exhibits matches its source. The other two point at something real but say more than the documents do.

What the better aviation study found

There is a stronger aviation study, and its result is more useful to SREs than the one the essay cites. In 2014 Stephen Casner and colleagues put 16 airline pilots in a Boeing 747-400 simulator, varied how much automation they could use, graded their flying, and asked them what they were thinking about as they flew. The physical skills held up: “We found pilots’ instrument scanning and manual control skills to be mostly intact, even when pilots reported that they were infrequently practiced.” The trouble came when pilots had to do the thinking that automation normally does for them. That meant tracking the aircraft’s position without a map display, deciding which navigation step came next, and recognising instrument failures. There the authors “observed more frequent and significant problems.” Performance on those tasks was tied to how often pilots reported thoughts unrelated to the task while the automation flew.

Which manual flying skills held up and which did not Two panels from Casner and colleagues, 2014, a study of 16 airline pilots in a Boeing 747-400 simulator. Left panel, mostly intact: instrument scanning and manual control of the aircraft, even when pilots said they rarely practiced them. Right panel, more frequent and significant problems: tracking the aircraft's position without a map display, deciding which navigational steps come next, and recognising instrument system failures. A footnote says performance on the cognitive tasks was associated with how often pilots had task-unrelated thoughts while automation flew. 16 AIRLINE PILOTS · BOEING 747-400 SIMULATOR · CASNER ET AL. 2014 Hands: mostly intact Head: more frequent, significant problems Instrument scanning Manual control of the aircraft Held up even when pilots reported practicing them infrequently Tracking position without a map display Deciding which navigational step is next Recognising instrument system failures Linked to how often pilots had task-unrelated thoughts on autopilot Performance graded across routine and non-routine scenarios at varied levels of automation.
The skills that faded were the cognitive ones, the work the automation had been doing for the pilot.

An SRE has almost no stick-and-rudder skill to lose. Typing a query or restarting a pod is the hands part. Almost everything else is in the right-hand panel: keeping a picture of where the system is, deciding what to check next, noticing that a signal is wrong rather than alarming. If Casner’s result carries over at all, AI triage threatens exactly the part of the job that is hardest to rebuild. The paper’s conclusion also points to a lever: retention “may depend on the degree to which pilots remain actively engaged in supervising the automation.” That fits the essay’s “human-in-the-loop, not human-on-the-side” recommendation better than its own sources do.

Two smaller findings worth keeping

The essay says the danger is that “you don’t feel it happening.” One laboratory study suggests the opposite under at least some conditions. Juergen Sauer and Alain Chavaillaz tested 24 people on a simulated process-control task, then tested them again eight months later. Afterwards trust in the automation went up and self-confidence went down. Diagnostic speed slowed, but diagnostic accuracy held. Their summary was that “subjective measures were more sensitive to the impact of skill lay-off than objective behavioural measures.” That was a complete lay-off, not months of working alongside an assistant, so it doesn’t settle the question. It does mean the essay’s claim that people can’t feel the decline is an assumption, not a finding.

On remedies, the essay proposes “deliberate inefficiency,” practice incidents worked by hand. Annette Kluge and Barbara Frank compared four kinds of refresher training in two experiments of 68 people each, on a simulated plant start-up procedure. Hands-on practice and skill demonstration supported skill retention. Symbolic rehearsal and a procedural knowledge test supported knowledge retention, but the people in those groups still showed moderate skill decay. For an SRE team, a quiz on the runbook won’t replace working a real or replayed incident.

Where that leaves the argument

Bainbridge’s point from 1983 is the strongest part of the essay, and it needs no help from shaky citations: automation leaves the operator with the rare, hard cases and takes away the routine practice that would prepare them. The newer evidence is thinner than the essay suggests, and the best of it says the cognitive skills fade first and that staying engaged protects them. For SRE teams that is a narrower and more useful claim than “AI is deskilling us”: the work to protect is the diagnosis, and the way to protect it is to keep people working incidents by hand rather than only approving what the assistant suggests.

References

  1. Elizabeth (SigNoz) (2026). AI Isn’t Replacing SREs. It’s Deskilling Them. Observability Real Talk, 28 February 2026.
  2. Budzyń, K., Romańczyk, M., Kitala, D. et al. (2025). Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. The Lancet Gastroenterology & Hepatology 10(10), 896–903. Published online 12 August 2025.
  3. Behrend, J. & Lafargue, C. (2019). The influence of automation (FD on vs. off) on pilots’ skills: an eye-tracking study. Flight Safety Foundation, International Air Safety Summit 2019.
  4. Federal Aviation Administration (2013). SAFO 13002: Manual Flight Operations. Flight Standards Service, 4 January 2013.
  5. Bainbridge, L. (1983). Ironies of automation. Automatica 19(6), 775–779, November 1983.
  6. Casner, S. M., Geven, R. W., Recker, M. P. & Schooler, J. W. (2014). The Retention of Manual Flying Skills in the Automated Cockpit. Human Factors 56(8), 1506–1516. Published online 16 May 2014.
  7. Sauer, J. & Chavaillaz, A. (2017). The use of adaptable automation: Effects of extended skill lay-off and changes in system reliability. Applied Ergonomics 58, 471–481, January 2017.
  8. Kluge, A. & Frank, B. (2014). Counteracting skill decay: four refresher interventions and their effect on skill and knowledge retention in a simulated process control task. Ergonomics 57(2), 175–190.