AI "Panel of Experts" Outdiagnoses Generalist Physicians in a Text-Only, Stepwise Test

In a study published in Science, researchers at Microsoft describe an AI system that works through a case the way a clinician does — asking questions, ordering tests, and revising its thinking as findings come back — rather than reading a fully written-up vignette. On a new interactive benchmark, the system, called DxO (Diagnostic Orchestrator), reached the correct diagnosis in 83.2% of difficult clinicopathological conference cases and 80.3% of emergency department admissions, versus 36.1% and 46.8% for the generalist physicians who took the same test. The work was led by senior author Eric Horvitz at Microsoft, with first author Harsha Nori.

Most medical AI benchmarks hand a model the chief complaint, history, exam findings, and lab results all at once. That compresses diagnosis into a single guess and can hide whether a system asks the right questions or over-tests. The team built SDBench, which turns 304 New England Journal of Medicine clinicopathological conference (CPC) cases and 325 MIMIC-IV-ED emergency department encounters into stepwise encounters. At each turn, the diagnostician — human or AI — can request history or exam details, order tests, or commit to a diagnosis. A "gatekeeper" agent returns only information the underlying record supports, and generates neutral results when data are missing so that gaps don't leak hints. A judge agent scored final diagnoses against the reference using a five-level rubric, counting answers for which management would be unchanged as correct.

DxO simulates a panel of clinical experts in complementary roles, keeps an explicit differential diagnosis, weighs the cost of each test, and aggregates results across independent runs. Testing costs were estimated from one major health system's price list.

The headline comparison is about the trade-off between accuracy and spending, not raw cost alone. DxO's unconstrained configuration achieved 83.2% accuracy while spending more per case than open-book physicians did on the CPC set ($5,431 versus $4,725) — with a configuration prioritizing diagnostic excellence reaching 84.5% at $8,945. The authors emphasize where each sits on the cost-accuracy "frontier" — the points at which nothing else is both more accurate and cheaper — and report DxO was more accurate at every spending level tested. When spending was capped, DxO kept a wide accuracy lead at lower cost: 80.9% at $2,241 and 79.6% at $1,303 on CPC cases, and 73.2% at $2,634 and 69.5% at $2,037 on emergency department cases. It also took a different route, averaging 25.5 questions before ordering its first test, compared with 5.6 for physicians.

Human performance came from 56 held-out CPC cases, yielding 764 closed-book and 662 open-book diagnoses. Physicians averaged 22.3% accuracy closed-book and 36.1% open-book; providing search and reference tools helped them but also raised their testing costs. The single best physician reached 51.8% on CPC cases. On emergency department cases, the best physician averaged 62.5% accuracy at $3,332 per case. Notably, DxO's accuracy held roughly steady across the two datasets, while physicians dropped sharply on the rare and atypical CPC cases.

Error analysis found DxO usually landed on a clinically adjacent answer at the wrong level of specificity, whereas physicians' misses were more often farther from the correct diagnosis. DxO's biggest edge over baseline models came on cases that hinge on careful history-taking, such as rare exposures and travel-related infections. Across all evaluated model families, DxO orchestration boosted diagnostic accuracy over minimally prompted baselines by an average of 7.9 percentage points on CPC cases and 4.4 points on emergency cases. When the system — developed using GPT-4.1 and evaluated with GPT-5 — was re-run with GPT-5.5 before publication without changing prompts or orchestration, its advantage over the baseline prompt actually increased.

The caveats matter. Both humans and AI interacted with a text "oracle" rather than real records, with no imaging, waveforms, or pathology slides, and none of the noise, ambiguity, and communication gaps of actual practice. The case mix skews toward diagnostically hard presentations and excludes benign or healthy encounters. The comparison is against individual generalist physicians, not specialists or multidisciplinary teams — the authors state plainly that these results do not establish superiority over those groups or over real-world care. Costs came from US pricing and ignore time-to-result, invasiveness, and patient discomfort, and the judge assessed diagnosis only, not treatment or outcomes.

What to watch next, the authors say, is the design of clinician-AI workflows that borrow these habits — explicit hypothesis tracking, adversarial self-critique, and cost-aware test selection — and the use of the same methods to build interactive teaching cases.

Reference: Nori H, Daswani M, Kelly C, et al. Sequential diagnosis with language models. Science 394(6822):423–428. DOI: 10.1126/science.aeh7930