Legal Advice, Beta Disclaimer Included
- Admitted
- 2026-08-12
- Source dataset
- Other / sample
- Source
- The Markup
- Failure mode
- Hallucination with side effects
- Agent system
- Single agent
- Tool access
- Read-only
- Industry
- General or other
Presenting Behaviour
A city government's official chatbot, built to help small business owners navigate local regulations, was asked a series of plain factual questions about labor and housing law. It said employers could take a cut of workers' tips (illegal), that landlords could refuse tenants paying with housing vouchers (illegal), and that businesses could go cashless (banned locally), each stated as settled fact. The errors were reported publicly and the chatbot stayed online, unchanged, for months afterward.
Case Replay
A reconstruction, not a simulation. This sequence is reconstructed from the public incident record and summarised in our own words. It is a static record of what was reported, and nothing here can be run or replayed live.
4 steps
- Factual questions. Users ask the city chatbot plain questions about labour and housing law.
- Wrong answers. It states illegal practices as settled fact, repeatedly and consistently. This is where the agent derailed.
- Public reporting. The errors are reported publicly.
- Disclaimer added. The city adds a not-legal-advice disclaimer but leaves the bot running unchanged.
Differential Diagnosis
Brief Psychotic Disorder (an isolated, self-correcting error), considered and ruled out. These wrong answers were reproducible across repeated, independent queries over an extended period, not a one-off slip the system moved past.
Malingering (optimizing for a measurable proxy instead of the real goal), considered. But there's no metric being gamed here; the chatbot wasn't being scored on anything that would reward wrong legal answers specifically.
Delusional Disorder (a fixed, incorrect belief about the task, maintained with consistency over an extended period despite the surrounding system otherwise functioning), most consistent with the pattern: the same category of wrong legal claims recurred reliably across many separate conversations over months, evidence of a settled and durable error rather than a passing one.
Working Diagnosis
Delusional Disorder: a stable, wrong model of local law that the chatbot applied consistently across many independent conversations, unchanged by the fact that its answers directly contradicted the regulations it was meant to help people follow.
Discussion
A wrong answer that never repeats is a nuisance; a wrong answer that repeats identically for months, from an official government source, is something people build real decisions on. The city's response (adding a disclaimer that its own official chatbot's answers shouldn't be used as legal advice, while leaving the chatbot running) illustrates the actual safeguard that was missing: a wrong, high-stakes answer needs to be corrected at the source, not disclaimed around.
Mitigation Strategy
Correct wrong, high-stakes answers at the source rather than disclaiming around them; test the bot against the regulations it is meant to explain.
Governance Guardrails
Pre-launch accuracy testing on legal topics; a takedown or restriction trigger once errors are reported; grounding in the actual regulation text.
Typical Presentation in Human Patients
General clinical background, included for context. Not medical advice, and not a diagnosis of any individual.
People with delusional disorder hold one or more fixed, false beliefs for a month or longer, maintained with full conviction despite clear evidence to the contrary. Unlike more severe psychotic conditions, day-to-day functioning and behaviour outside the delusion's specific theme usually remain largely intact.
Source Note
The Markup
https://themarkup.org/artificial-intelligence/2024/04/02/malfunctioning-nyc-ai-chatbot-still-active-despite-widespread-evidence-its-encouraging-illegal-behavior