Twice, Four Times or 1.7 Times: The Multiple Depends on Which Doctors

On 22 October Science published “Sequential diagnosis with language models,” the peer-reviewed version of a preprint Microsoft AI posted in June 2025. The headline that reached Hacker News the next day was Gizmodo’s: Microsoft’s new AI is “more than twice as good at diagnosing diseases as doctors.” Microsoft’s own post, published the same day as the paper, gives three multiples in two paragraphs. The system, MAI-DxO, is “more than twice as accurate as physicians using textbooks and the internet” and “nearly four times as accurate when they diagnose without those resources” on hard cases, and it “is 1.7 times more accurate than physicians” on ordinary emergency department ones.

All three are correct, and each describes a different comparison. Which one a reader hears depends on which physicians and which cases the sentence leaves out.

Five numbers behind three multiples

The Science abstract, which I read through the journal’s Crossref record because the article page refused my requests, gives the accuracies. On New England Journal of Medicine clinicopathological conference cases the system reached 83.2%, against 36.1% for generalist physicians. On MIMIC-IV-ED, a research dataset of emergency department admissions, it reached 80.3% against 46.8%. Microsoft’s post adds the arm without references: physicians who could not consult textbooks or the internet scored 22% on the NEJM cases.

The arithmetic is short. 83.2 divided by 36.1 is 2.3. 83 divided by 22 is 3.8. 80.3 divided by 46.8 is 1.7. “Twice as good” is the middle comparison of the three: references allowed, cases picked for difficulty.

MAI-DxO and physician diagnostic accuracy in three comparisons Horizontal bars on a 0 to 100 percent scale. NEJM case conference cases: MAI-DxO 83.2 percent, physicians with references 36.1 percent, physicians without references 22 percent. MIMIC-IV-ED emergency admissions: MAI-DxO 80.3 percent, physicians 46.8 percent. The resulting multiples are 2.3, about 3.8 and 1.7. DIAGNOSTIC ACCURACY · SCIENCE, 22 OCT 2026, AND MICROSOFT AI NEJM case conferences (304 hard cases) MIMIC-IV-ED emergency admissions MAI-DxO Physicians, with references Physicians, no references MAI-DxO Physicians 83.2% — Science abstract, 22 Oct 2026 36.1% — Science abstract, 22 Oct 2026 22% — Microsoft AI post, 22 Oct 2026 80.3% — Science abstract, 22 Oct 2026 46.8% — Science abstract, 22 Oct 2026 83.2% 36.1% · 2.3x 22% · about 3.8x 80.3% 46.8% · 1.7x Multiples are MAI-DxO divided by the physician bar beside it. Benchmark results, not patients.
One system, three multiples. The headline "twice" uses physicians who were allowed references; drop the references and the gap nearly doubles, switch to ordinary emergency cases and it shrinks to 1.7.

Who the physicians were

The Science paper is behind a page I could not load, so for the physician arm I went to the June 2025 preprint, which describes it in full. The comparison group was 21 physicians practising in the US or UK, with a median of 12 years of experience. Seventeen were primary care physicians and four were hospital generalists. They were told not to use search engines, language models or other online medical information, and together they made 764 diagnoses on the 56 most recent NEJM cases, published in 2024 and 2025 and held back as a test set. Their mean accuracy was 19.9%. That is the “four times” of the preprint’s abstract, which compared roughly 80% for the system against “the 20% average of generalist physicians.”

The 21 physicians in the preprint's comparison arm Twenty-one squares, one per physician: 17 filled squares for primary care physicians and 4 outlined squares for hospital generalists. Below: median 12 years of experience, practising in the US or UK, 764 diagnoses on 56 held-out NEJM cases from 2024 and 2025, mean accuracy 19.9 percent, with no search engines or language models allowed. PHYSICIAN ARM · ARXIV PREPRINT, JUNE 2025 17 primary care physicians 4 hospital generalists Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Primary care physician — arXiv:2506.22405 Hospital generalist — arXiv:2506.22405 Hospital generalist — arXiv:2506.22405 Hospital generalist — arXiv:2506.22405 Hospital generalist — arXiv:2506.22405 12 years 764 diagnoses 56 cases 19.9% median experience US or UK practice NEJM, 2024 and 2025 mean accuracy No search engines, language models or other online medical information. All generalists.
The "four times" baseline is this group: generalists without references, on the hardest published cases. The Science version adds an arm with references, which is where the smaller multiple comes from.

The paper names the two choices that set that number. It “recruited medical generalists only,” and notes that real cases like these may be referred to specialists. It also says that barring search tools does not reflect everyday practice. The Science version adds the arm where physicians had textbooks and the internet, and that arm moves the NEJM physician score from about 20% to 36.1%. It is the more realistic comparison, and it is the one behind the smaller headline multiple.

What the benchmark measures

The benchmark, SDBench, turns each NEJM case into a sequence. The diagnostician starts with a short presentation and has to ask for history, examination findings and tests one at a time. In the preprint those answers come from a gatekeeper model, o4-mini, which holds the full case file, and a judge model, o3, grades the final diagnosis against the published one using a rubric written by physicians. Cost comes from mapping each requested test to a billing code and pricing it from a 2023 price list published by a large US health system, plus a flat $300 per physician visit.

How a case runs in the Sequential Diagnosis Benchmark A flow diagram. The diagnostician, either MAI-DxO or a physician, asks a gatekeeper model, o4-mini, which holds the full case file and answers one request at a time. Each requested test is priced from a 2023 price list published by a large US health system, plus 300 dollars per physician visit. The final diagnosis goes to a judge model, o3, which grades it against the published diagnosis using a rubric written by physicians. ONE CASE IN SDBENCH · AS DESCRIBED IN THE PREPRINT Diagnostician Gatekeeper · o4-mini Cost estimate Judge · o3 MAI-DxO or a physician holds the full case file and answers one request at a time each test priced from a 2023 US health-system price list, plus $300 per physician visit physician-written rubric, vs the published diagnosis final diagnosis Every answer and every grade comes from a model. No patient is involved at any step.
The diagnostician never sees the case file. It sees what the gatekeeper returns, pays an estimated price for each test, and is graded by a second model against the NEJM write-up.

So the “half or even one third of the cost” claim is an estimate over simulated cases, and no bills were paid. The preprint is direct about the other limits: NEJM cases are chosen for difficulty, so the benchmark cannot measure false positive rates or reflect a real case mix. The Science abstract has the same caveat in one clause, describing the findings as “obtained in a controlled benchmark setting.”

What would settle it

Microsoft’s post says the gains “persisted – and improved” when it reran the evaluation with newer models, GPT-5.5 among them, and that the next step is a prospective study at Beth Israel Deaconess Medical Center in Boston, measured against physicians’ independent assessments. That is the right test, and none of the three multiples predicts its result. A prospective study sees live patients rather than written-up cases, and none of the numbers above was measured on live patients.

My one-line version of the Science paper is narrower than any of the headlines. On a sequential benchmark built from hard published cases, an orchestrated language model found the published diagnosis far more often than generalists did, by 2.3 times when the generalists had their references and 1.7 times on ordinary emergency cases. That is a strong result for hard cases worked one question at a time. The Boston study will report how it does on patients, and it has not reported yet.

References

  1. Nori, H., Daswani, M., Kelly, C. et al. (2026). Sequential diagnosis with language models. Science 394(6822), 423–428, 22 October 2026. Abstract read via the Crossref record.
  2. MAI Health Team (2026). Building medical superintelligence. Microsoft AI, 22 October 2026.
  3. Nori, H., Daswani, M., Kelly, C. et al. (2025). Sequential Diagnosis with Language Models. arXiv:2506.22405, v1 27 June 2025, v2 2 July 2025.
  4. Gizmodo (2026). Microsoft Says Its New AI Is More Than Twice as Good at Diagnosing Diseases as Doctors. 23 October 2026.
  5. Gautam Parab (2026). Solved on the Set It Was Scored On. 19 October 2026.