Microsoft just wrapped up a huge week in medical AI. A new paper in Science showed MAI-DxO is more accurate than doctors, Microsoft AI charted its course to medical superintelligence, and the same team published a framework for what that actually looks like.
There’s a long road ahead. The Science paper formalizes SDBench, which turns 304 NEJM cases and 325 MIMIC-IV emergency admissions into step-by-step encounters where the diagnostician asks questions, orders priced tests, and decides when to commit.
- The setup is meant to emulate the day-to-day realities of how doctors operate, which means gradually gathering information by asking a whole lot of questions.
MAI-DxO plays a panel of virtual physicians that debate each case.
- On complex NEJM cases, MAI-DxO scored 83% on diagnostic accuracy, versus 36% for physicians using references (and 22% without).
- On ED cases, MAI-DxO notched 80%, again outshining the physicians at 47%.
- The AI asked 25.5 questions before its first test. Physicians asked 5.6.
That might sound like medical superintelligence. Microsoft AI’s companion post frames the paper as a step toward that AI holy grail, but confirms there’s still a long road ahead.
MAI CEO Mustafa Suleyman predicted medical superintelligence was two to three years away last summer, and is now taking steps to get us there even faster:
- Microsoft is open-sourcing SDBench for select NEJM cases, with the Gatekeeper, Judge, and cost code included so others can test their models against it.
- Microsoft is running a study at Beth Israel Deaconess Medical Center to have patients interact with MAI-DxO directly. That’s the same BIDMC working with Google’s AMIE team.
- Microsoft is folding its health scientists and the Dragon Copilot and PowerScribe teams into Microsoft AI, alongside its Mayo Clinic frontier model collaboration.
Now the fine print, also written by Microsoft. A new BMJ Digital Health & AI paper defines medical superintelligence as surpassing the strongest human-led health systems across the full arc of care, then lists six dimensions that would entail:
- medical knowledge, diagnostic reasoning and triage, therapeutic decisions, longitudinal context, clinical coordination, and alignment with human values.
MAI-DxO only covers about two of the six. The other four involve sensing what patients actually want, treating them over years, and coordinating humans to make it happen
The Takeaway
Microsoft definitely gets credit for publishing its to-do list alongside its victory lap. Diagnosing NEJM cases is a nice box to check, but the remaining boxes are the ones without answer keys.
