A sepsis prediction model was run silently across nine hospitals (two quaternary academic centers, five community hospitals, two critical access) for five and a half months in 2024, generating a score every fifteen minutes for 198,494 adult encounters. Nobody saw the output. The run existed to score the same predictions against three different official definitions of sepsis and see what happened.
What happened is that the model has an AUROC of 0.94, or 0.89, or 0.85, depending on which definition you write in the caption.
The paper is Dutta and colleagues in JAMA Network Open, published 7 April 2026 and open access. The model is Epic’s Early Detection of Sepsis Model version 2 — not the widely criticized original, and not the vendor’s shipped weights either, but a gradient-boosted tree ensemble retrained locally on this health system’s own data, optimized against the Sepsis-3 outcome. The three definitions are CMS’s SEP-1, the Sepsis-3 consensus criteria, and the CDC’s Adult Sepsis Event. They are all in current use. They disagree about which patients have sepsis: in this cohort SEP-1 finds 2,366 encounters (1.2%), Adult Sepsis Event finds 3,881 (2.0%), and Sepsis-3 finds 5,832 (2.9%).
SEP-1 — the narrowest definition, the one that finds the fewest patients — is the one that scores 0.94.
The second metric ranks them backwards. On area under the precision-recall curve, Sepsis-3 comes first at 0.24, SEP-1 drops to 0.16, and Adult Sepsis Event lands last at 0.11. So the definition that produces the best-looking model on one standard chart produces the second-worst on the other, and the ordering is not close to stable.
That inversion is not a paradox and the authors do not present it as one. It is arithmetic, and they say so plainly: “Variations in incidence per sepsis outcome definition affect the PPV and therefore AUPRC, but the AUROC is incidence independent.” AUROC asks whether the model ranks a random septic patient above a random non-septic one, which does not depend on how many septic patients there are. AUPRC asks what fraction of flagged patients are really sick, which depends on it enormously. Halve the prevalence and you have roughly halved the ceiling on precision without touching a single model weight. I have written before about what happens when the denominator does the work in a safety statistic; this is the same failure wearing a different coat. The number moves, the model does not.
If that were the whole finding it would be a useful methods note and not much more. Anyone can be told to report AUPRC alongside AUROC. A sensitivity analysis buried further down matters more.
A definition does more than set a threshold on physiology. Two of these three definitions are partly operational: they require that somebody ordered blood cultures, or started antibiotics, within a specified window. The label therefore encodes a clinician’s decision. If the model has access to the same chart the clinician was reading, and the outcome is defined by what that clinician then did, some portion of the model’s apparent foresight is not foresight at all.
So the authors moved the clock. They re-scored predictions using only what was knowable before the sepsis-related clinical actions: the ordering of lactic acid, blood cultures, or intravenous antibiotics.
The answer depends sharply on which action you pick. Censoring to before blood cultures and antibiotics cost SEP-1 a little, and moved Sepsis-3 and Adult Sepsis Event essentially not at all. But censoring the Sepsis-3 predictions to before the first lactic acid order dropped AUROC from 0.89 to 0.75, and raised AUPRC from 0.24 to 0.38.
Both numbers moved, in opposite directions, for the same reason as before: censoring changes the cohort and its base rate along with it. So I would not read the AUPRC rise as good news, and I would not read the fourteen-point AUROC fall as a clean measure of borrowed foresight either. The authors do not. Their discussion reports “no significant degradation of model performance on any of the 3 sepsis outcome definitions when evaluated only on predictions prior to clinical recognition of sepsis.”
That sentence and the 0.75 sit in the same paper. The gap between them is the part worth noticing: one cutpoint out of three moves the headline number by fourteen points, and whether that counts as degradation depends on which clinical action you treat as the moment somebody began to suspect sepsis. Which is another definitional choice, not a measurement.
The hazard is not new, and this paper does not settle it. But the mechanism it points at shows up plainly in the label data. A Dutch multicentre study published in eClinicalMedicine in 2025, older than the window I normally hold evidence to and offered here as background, found that in the emergency department, 92.9% of sepsis patients received antibiotics against 31.3% of non-sepsis patients, which makes “antibiotics were ordered” an informative signal. On the wards, antibiotics were ordered at similar rates in septic and non-septic patients, and their label model fell apart, topping out at an F1 of 52.2%. The treatment signal carries real information exactly where treatment decisions are decisive, and none where they are routine.
Which sets up the finding that has the most direct bearing on where you would actually install the thing. Holding the definition fixed at Sepsis-3 and the prediction horizon at eight hours, discrimination falls monotonically with care setting: 0.90 in the emergency department, 0.82 on the inpatient wards, 0.76 in the ICU.
The authors’ explanation is risk compression, and it is convincing. Their words: ward and ICU patients “have a higher and more homogeneous baseline risk, with many exhibiting abnormal physiologic parameters or early organ dysfunction, compressing the range of predicted risk.” A model that separates the deteriorating patient from the stable one has an easy job in a waiting room full of mostly-well people and a hard one in a unit where everybody’s vitals are already deranged. Ranking is only impressive when there is a spread to rank.
The awkward implication is that the score is highest in the setting with the most time to act and lowest in the setting where the consequences are worst. A single AUROC printed on a slide is an average over that gradient, weighted by wherever the vendor’s evaluation cohort happened to come from.
A companion paper from a different group, also in JAMA Network Open, published 27 February 2026, makes the point across institutions instead of across definitions. Wong and colleagues validated the same updated Epic model prospectively on 227,091 inpatient encounters at four large US health systems. Encounter-level AUROC ranged from 0.82 to 0.92 depending on the site. At a threshold tuned for 60% sensitivity, positive predictive value ran from 0.13 to 0.26, and the number of patients a clinician must evaluate to find one true case ran from 21 to 35 on a twelve-hour horizon, and from 24 to 69 at four hours. Their conclusion names “high institutional variability, low positive predictive value, and high alert burden,” and recommends local validation before deployment, which is a polite way of saying the vendor’s number is not your number.
To be fair to the model, it beat its own predecessor at every site and threshold: version 1 scored 0.65 to 0.84 with a PPV of 0.07 to 0.14. The revision is real. It is the reporting that has not caught up.
For historical scale, a 2023 study in JAMA Internal Medicine of the original model across nine networked hospitals found C-statistics from 0.55 to 0.73, against a developer-claimed range of 0.76 to 0.83, with performance correlating negatively with each hospital’s sepsis incidence. And in a 2021 paper covering 24 hospitals, the proportion of patients generating a sepsis alert each day more than doubled, from 9% to 21%, across the three weeks either side of each health system’s first COVID-19 case, with total daily alerts up 43% even as the census fell 35%. The authors attribute much of that to cancelled elective surgery leaving a sicker inpatient population, and they note plainly that they did not evaluate the model’s accuracy. Which is its own kind of finding: the alarm burden on the ward doubled and nobody was in a position to say whether the model had got better or worse. Both papers predate my usual evidence window and are cited here as context, not proof, but they establish that the spread is old news.
None of this would matter if the definitions were carved into the disease. They are not. They were drawn by committees, and the drawing has moved.
The clearest demonstration I know is a 2015 paper in the New England Journal of Medicine by Kaukonen and colleagues, which examined 1,171,797 ICU admissions across 172 units in Australia and New Zealand. Of the 109,663 patients with both infection and organ failure, 13,278 — one in eight — failed to meet the then-required two SIRS criteria, and so by the definition in force were not cases of severe sepsis. Their mortality fell from 27.7% to 9.3% over the study period, on a trend statistically indistinguishable from the patients who did qualify. And mortality rose smoothly with each additional SIRS criterion met, with no discontinuity at two. The threshold that decided who counted did not correspond to any jump in the risk of dying. It was a convention, and a year later Sepsis-3 retired it.
A model trained and scored on a convention inherits the convention.
The practical reading of the 2026 paper is short. An AUROC for a sepsis model is uninterpretable without four accompanying facts: which outcome definition, at what prediction time relative to the clinician’s first action, in which care setting, and at what threshold, with the resulting positive predictive value and number-needed-to-evaluate stated in the same breath. Drop any one of those and the number can be moved by several points without retraining anything, which is roughly the situation benchmark scores are in more generally.
There is a 2024 result, again outside my window and offered as background, that makes this concrete: Cohen and colleagues in Scientific Reports varied only the onset-time convention within Sepsis-3 — not the definition, just when the clock starts — and moved AUROC by around six points, more than the spread between the competing model architectures they were comparing. Under one convention the best model scored 0.829; under another, the worst-performing method scored 0.844.
If the choice of label moves the leaderboard further than the choice of model, then a leaderboard that does not print its label is not reporting a result. It is reporting a decision that somebody made upstream, and did not write down.
References