The Referral That Was Never Made
- Admitted
- 2026-05-12
- Source dataset
- Other / sample
- Failure mode
- Hallucination with side effects
- Agent system
- Single agent
- Tool access
- Write access
- Industry
- Healthcare
Presenting Behaviour
Ontario's auditor general reviewed AI-powered medical transcription tools that had been approved for use by doctors across the province. The audit found that most of the approved tools produced clinical notes containing fabricated content: invented referrals for tests or therapy that had never actually been discussed, incorrectly transcribed prescription medication names, and summaries that missed key details from mental health encounters. The tools had received official approval to be used in real patient care despite these problems surfacing during evaluation.
Case Replay
A reconstruction, not a simulation. This sequence is reconstructed from the public incident record and summarised in our own words. It is a static record of what was reported, and nothing here can be run or replayed live.
4 steps
- Tools approved. AI medical scribes are approved for use by doctors.
- Audit. The auditor general reviews the tools. This is where the agent derailed.
- Fabrication found. Notes contain invented referrals, wrong medication names and missed mental-health details.
- Already visible. The problems were visible in evaluation before approval.
Differential Diagnosis
Apraxia (transcription is the tool's job, and it fumbled it), considered, but transcription errors alone would just mean missing or garbled content; the more troubling finding is content that was invented outright, like referrals that were never discussed, which isn't a failed transcription of something real.
Delusional Disorder (one sustained false belief carried across a task), doesn't fit either; each fabricated detail was a separate, unrelated invention rather than one consistent false premise running through an entire session.
Brief Psychotic Disorder, most consistent with the pattern. Each fabrication (a referral, a medication name, a diagnosis) was a discrete, confidently stated clinical fact that had no basis in the actual encounter, generated and inserted into an otherwise normal-looking note.
Working Diagnosis
Brief Psychotic Disorder: the scribe tools generated specific, plausible-sounding clinical details that were never actually said or done during the patient encounter, each one a confident fabrication rather than part of a broader consistent misunderstanding.
Discussion
A fabricated referral in a medical record isn't a cosmetic error, it can change what care a patient receives or misinform the next clinician who reads the chart. The audit's most important finding may not be the tools' error rate at all, but that procurement approved them for real patient use despite fabrication problems that were already visible during evaluation; catching this kind of failure has to happen before deployment, not after.
Mitigation Strategy
Catch fabrication before deployment, so procurement does not approve tools whose problems are visible in evaluation.
Governance Guardrails
Pre-deployment accuracy evaluation; clinician review of generated notes; procurement criteria tied to fabrication rates.
Typical Presentation in Human Patients
General clinical background, included for context. Not medical advice, and not a diagnosis of any individual.
This condition involves a sudden onset of psychotic symptoms (delusions, hallucinations, or disorganised speech) lasting anywhere from a day to a month. It's frequently triggered by a significant stressor, and the person typically returns fully to their prior level of functioning once the episode resolves.
Source Note
OECD.AI Incidents Monitor (citing Ontario Auditor General)
https://oecd.ai/en/incidents/2026-05-12-76df