Google's AMIE chatbot matches doctors in a hospital triage study

Google published results in The Lancet from a real-world hospital trial of AMIE, its research diagnostic chatbot, saying it was the company’s first paper to appear in the journal’s main publication. The study ran at Beth Israel Deaconess Medical Center’s ambulatory primary care clinic, where 98 patients consulted AMIE ahead of urgent care visits while supervising physicians monitored the conversations in real time.

Google reported that AMIE’s differential diagnoses, generated from the pre-visit conversation, matched the treating clinicians’ eventual final diagnoses 90% of the time. Clinicians told researchers the AI-generated summaries helped them prepare for the visit in about three-quarters of cases and influenced their approach to care in more than half. The company said no conversation had to be interrupted on safety grounds during the trial.

The study is a step beyond AMIE’s earlier evaluations, which largely tested the model in simulated consultations or against written case vignettes rather than in a live clinic with real patients and working physicians. Putting the system in front of patients awaiting actual appointments, with a doctor able to intervene, is a different kind of evidence than a benchmark score, though a sample of 98 patients at a single academic medical centre is small, and the result speaks to diagnostic agreement rather than to outcomes such as whether care improved or errors fell.

The paper’s authors, including Google research lead Mike Schaekermann, said explicitly that larger clinical trials are needed before AI systems like AMIE could be used directly with patients at scale. The publication keeps AMIE in a research and pre-visit support role rather than establishing it as a deployed clinical tool, but it gives Google a peer-reviewed data point to point to as it and other labs continue pushing AI into medical settings.