Microsoft’s AI Compared With Doctors: The Study’s Measure Explained

🔍 Read the full analysis: Microsoft’s AI Compared With Doctors: The Study’s Measure Explained on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic system scored higher than generalist physicians on two sets of challenging medical case records in a study published October 22 in Science. The paper reports 83.2% accuracy across 304 complex cases, compared with 36.1% for physicians using textbooks and the internet, but the results come from a simulated, text-only benchmark—not care delivered to real patients.

The study, titled “Sequential diagnosis with language models,” tested whether an AI system could gather information and select tests over several rounds before reaching a diagnosis. Its Sequential Diagnosis Benchmark begins each case with a short summary. The system or physician must request further information from a model holding the full case record, with a cost assigned to each test. The benchmark evaluates both the diagnosis and the estimated expense of reaching it.

On the 304 New England Journal of Medicine case records, the paper reports 83.2% diagnostic accuracy for MAI-DxO. Physicians scored 36.1% when allowed textbooks and the internet, and 22.3% without those resources. In a separate set of 325 emergency department cases, the reported scores were 80.3% for the system and 46.8% for physicians. Microsoft presented these results as evidence of a substantial performance advantage on the tested cases.

MAI-DxO coordinates a language model through a panel of virtual physician agents that propose diagnoses, request information or tests, and critique one another. The study’s reported results are Microsoft’s own; all 15 paper authors are listed with Microsoft affiliations. Publication in a peer-reviewed journal indicates that the work underwent review, but it does not establish that independent research teams have reproduced the findings.

What the Scores Mean for Care

The benchmark addresses a real challenge in diagnosis: clinicians do not simply receive a complete case file and select an answer. They decide what to ask, which test to order, and when to stop. A system that can gather information selectively and account for costs could, in principle, help clinicians work through complex cases. The paper tests that process in a controlled setting, rather than only testing whether a model can recognize a diagnosis when all the facts are supplied.

But the score difference does not show that MAI-DxO is ready to replace doctors or improve patient outcomes. The comparison was with individual primary care doctors and hospital generalists, not specialist teams or colleagues consulting together. The cases were selected for difficulty, and no real patients received care from the system in this evaluation. The findings are relevant to further clinical research, but do not by themselves establish safety, effectiveness, or value in a hospital or clinic.

How the Benchmark Was Built

The published study follows an earlier, non-peer-reviewed version posted on arXiv in June 2025. That preprint reported a larger gap, using an earlier version of the underlying model. The October 2026 Science paper reports updated figures and includes a physician comparison with reference materials. The earlier preprint’s methods describe 21 U.S. and U.K. generalists, with a median of 12 years’ experience; the available source notes that some implementation details may differ in the final paper.

Several measurement choices affect how the scores should be read. In the earlier version, a model judged answers against the published diagnosis, and physician re-grading showed good but imperfect agreement. If a requested test result was missing from the record, a gatekeeper model generated a synthetic result. Test prices were estimates based on billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit charge. The paper’s authors described these as approximations. These methods make the benchmark possible, but they are not the same as measuring actual clinical spending or patient outcomes.

The earlier preprint also described a smaller physician evaluation using 56 held-out cases, while the AI was assessed on all 304. The October paper reports the headline comparison figures cited above. Because the full Science article is behind a paywall, the source account cautions that details taken from the preprint may not match every part of the final publication.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch

Limits Behind the Accuracy Figures

The study does not establish how MAI-DxO would perform with live patients, physical examinations, imaging, or the wider clinical record. Nor does it show whether its recommendations would change treatment, reduce diagnostic delays, avoid harm, or save money. The case sets contain difficult published cases rather than a typical mix of patients, including healthy people and cases with benign explanations.

The source account says the authors noted that this case selection meant the study could not measure false alarms. A system tested mainly on rare, difficult diagnoses may perform differently when deciding whether a common or harmless explanation is more likely. It is also not yet clear from the available information how the final study handled all methodological details described in the preprint, or whether independent groups will reproduce the results.

A Study Alongside Routine Care

Microsoft says a prospective study is planned at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with the system alongside their normal care. That setting could provide evidence unavailable from historical case records, including how the tool fits into clinical workflows and how its recommendations compare with routine care.

Details such as the study’s size, timeline, patient groups, outcome measures, and whether results will be independently evaluated are not provided in the source material. Until those results are available, the reported accuracy figures should be understood as performance on the stated benchmarks, not proof of clinical benefit.

Key Questions

What did Microsoft’s MAI-DxO study measure?

It measured whether the system could gather information, request tests, and reach diagnoses in simulated case records. The benchmark also assigned estimated costs to tests; it did not measure outcomes from treatment of real patients.

How did the system score against physicians?

The paper reports 83.2% accuracy for MAI-DxO on 304 complex cases, versus 36.1% for physicians using textbooks and the internet and 22.3% without those resources. On 325 emergency department cases, it reports 80.3% for the system and 46.8% for physicians.

Were the doctors specialists?

The comparison group consisted of generalist physicians, including primary care doctors and hospital generalists. The source account says the study did not compare MAI-DxO with specialists or teams of doctors.

Does the paper prove the system is safe for clinical use?

No. The results are from a peer-reviewed benchmark using historical, text-only cases. They do not establish safety, improved patient outcomes, or effectiveness in everyday clinical care.

What is the next planned test?

A prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would interact with the system alongside their normal care. The source material does not provide the study’s timeline or detailed design.

Source: ThorstenMeyerAI.com