A patient needs a reliable diagnosis and a clear path to treatment. For healthcare leaders, Microsoft’s latest medical AI research raises a practical question: how should an organisation evaluate a system that gathers evidence and makes decisions over several steps?
My view is that MAI-DxO deserves attention for its decision process as well as its accuracy. CEOs and CTOs should study that process while keeping a clear boundary between research performance and demonstrated clinical value.
The study, Sequential diagnosis with language models, appeared in Science on 22 October 2026. MAI-DxO paired with GPT-5 achieved 83.2% diagnostic accuracy on 304 difficult NEJM cases. Generalist physicians using reference resources achieved 36.1%, compared with 22.3% without them. On 325 emergency-department admission cases, the system achieved 80.3%, compared with 46.8% for physicians. These are results from controlled evaluations of case records. [1, 2]
Those physician percentages describe performance on the study tasks. They should never be presented as the general accuracy of doctors in everyday practice. An overall benchmark score also leaves a crucial question unanswered: which errors could cause the greatest harm?
MAI-DxO coordinates a virtual panel that proposes diagnoses, challenges assumptions and selects questions or tests. Its assessment changes as information arrives. The benchmark measures the route to a diagnosis, including estimated testing costs. Microsoft also evaluated configurations with stronger cost constraints. [2]
For enterprise leaders, the useful design lesson is to separate proposing an answer from reviewing its evidence. A reviewer should be able to identify what is missing, challenge the leading explanation and request further information before a consequential action proceeds.
Several AI roles can still repeat the same mistake. Agreement between agents should therefore be tested against external evidence. Adding debate creates another component to evaluate; it does not establish independent verification.
I would compare systems along three dimensions. Compare a final-answer score with the quality of the full decision process. Compare estimated task costs with the total cost of delivering the service. Compare performance on historical records with outcomes observed during prospective use. Each comparison answers a different investment question.
For example, an inexpensive recommendation may create substantial review work. A correct answer may arrive too late to help. A system that performs well on complete records may struggle when important information is unavailable. Leaders need an evaluation that makes these differences visible.
SDBench uses a gatekeeper model to provide requested information and a judge model to assess diagnoses. The gatekeeper can generate case-consistent findings when a requested result is absent from the record. These mechanisms make interactive evaluation possible, but they also introduce assumptions that need scrutiny. [3]
That matters because the evaluation environment influences the measured result. A hospital must establish how a system behaves with incomplete information, conflicting accounts and operational delays. Diagnostic accuracy alone cannot establish patient safety, treatment quality or the effectiveness of a clinical team.
Microsoft has announced a prospective study with Beth Israel Deaconess Medical Center in which patients will interact with the system and performance will be compared with physicians’ independent assessments. This is planned validation, rather than evidence of improved patient outcomes already achieved. [2]
CEOs should define the intended benefit before funding deployment. In a healthcare pilot, that might be faster recognition of a specific condition or better support for reviewing difficult cases. Name the clinical owner, agree the outcome measures and establish the conditions for stopping the pilot. Avoid making a staffing decision from a benchmark comparison.
CTOs should build an evaluation around the intended workflow. Compare the existing process, AI working alone in a controlled setting, and clinicians supported by AI. Use cases representative of the proposed service. Examine serious missed diagnoses, unnecessary investigations, time to decision and the burden of reviewing recommendations. Report results for relevant patient groups as well as the overall average.
Start with retrospective evaluation using appropriately governed data. If the evidence supports progression, consider a supervised prospective study with clinical leadership and the required approvals. Keep authority to order tests or change treatment explicitly assigned. Define when the system must defer and how a clinician can override it.
Require traceable evidence for recommendations, controlled access to patient information and a record of the model and workflow version used. Repeat the relevant evaluations when either changes. A successful pilot needs an operating process that can detect deterioration and support withdrawal.
Finance leaders should distinguish estimated testing costs from realised savings. Include integration, clinical review, monitoring and ongoing evaluation in the business case. Any saving should be assessed alongside the consequences of missed diagnoses and avoidable investigations.
The same evaluation discipline is useful when considering AI for insurance claims, procurement exceptions or incident investigation. Those applications need their own evidence. A medical benchmark cannot validate a financial or operational workflow.
MAI-DxO offers a reason to examine how AI asks for evidence, revises its position and decides when to stop. Enterprise value will depend on whether those behaviours improve a defined service under real operating conditions.
If you were approving a medical AI pilot, what evidence would you require before allowing its recommendations to influence patient care?
Think mathematically. Engineer intelligently. Lead strategically. Build responsibly. Scale globally. Create real business value through artificial intelligence.
[1] Nori, H. et al. (2026). Sequential diagnosis with language models. Science. https://doi.org/10.1126/science.aeh7930
[2] Microsoft AI (22 October 2026). Building Medical Superintelligence. https://microsoft.ai/news/building-medical-superintelligence/
[3] Microsoft. SDBench documentation. https://microsoft.github.io/sdbench/
