Your doctor’s AI notetaker may be making things up, Ontario audit finds

An audit in Ontario has revealed significant flaws in AI medical scribes recommended by the provincial government. These AI systems, designed to summarize patient conversations and diagnoses, frequently generated incorrect, incomplete, or fabricated information. The auditor general warned that such inaccuracies could lead to inadequate or harmful treatment plans, potentially jeopardizing patient health.

The audit reviewed transcription tests from 20 AI scribe vendors approved by the government. All vendors exhibited issues with accuracy or completeness. Specifically, nine hallucinated patient information, twelve recorded information incorrectly, and seventeen missed crucial details regarding mental health discussions.

Concerning examples highlighted in the report include AI scribes fabricating non-existent referrals for tests, incorrectly transcribing medication names, and omitting key details about mental health issues. These errors have direct negative implications for subsequent patient care.

Despite the critical importance of accuracy, this metric accounted for only about 4 percent of a vendor's overall evaluation score. This allowed AI scribes to meet approval thresholds even with zero scores in accuracy, while "domestic presence in Ontario" surprisingly held a 30 percent weighting.

Consequently, the auditor general concluded that these AI scribes were "not evaluated adequately." The report urged rigorous testing of AI scribe systems to ensure the quality of their generated notes and minimize inaccuracies. It also recommended that IT departments mandate doctors to confirm their review of the notes before committing them to patient logs.

While public sector health services are not obliged to use these AI scribes, the government's recommendation of systems with such glaring and potentially harmful flaws should prompt caution among doctors and patients alike.