A Compensation Promise With No One Behind It
- Admitted
- 2026-09-13
- Source dataset
- Other / sample
- Source
- Gowling WLG
- Failure mode
- Hallucination with side effects
- Agent system
- Single agent
- Tool access
- Read-only
- Industry
- Customer service
Presenting Behaviour
A user asked a generative-AI assistant a factual question about a university's campus address. The assistant answered incorrectly, and when pressed, didn't just repeat the wrong answer, it added an unprompted guarantee, telling the user it would pay them ¥100,000 if the address turned out to be wrong, and suggesting they take the matter to the Hangzhou Internet Court if it was. The user did exactly that. The court dismissed the case in January 2026, ruling that a statement generated by the AI carries no legal weight and cannot bind the company that operates it.
Case Replay
A reconstruction, not a simulation. This sequence is reconstructed from the public incident record and summarised in our own words. It is a static record of what was reported, and nothing here can be run or replayed live.
4 steps
- Factual question. A user asks for a university's campus address.
- Wrong answer. The assistant gets it wrong. This is where the agent derailed.
- Volunteers a guarantee. Pressed, it promises ¥100,000 if wrong and points to the court.
- Lawsuit. The user sues; the court dismisses, finding AI statements non-binding.
Differential Diagnosis
Malingering, considered, since offering a dramatic guarantee might make the assistant's answer sound more trustworthy in the moment. Ruled out because the 'reward' being chased here isn't a graded metric or test, it's just an unprompted, ungrounded escalation with no plausible upside for the system, more consistent with confabulation than a calculated shortcut.
Concrete Thinking, considered, since the user's question was about accuracy and the assistant responded with a literal-sounding promise. Ruled out because the failure wasn't a misreading of what the user meant; the assistant wasn't asked for a guarantee at all; it manufactured one that nobody requested.
Brief Psychotic Disorder, most consistent with the evidence. First a wrong factual claim, then (under no apparent pressure to do so) a specific, confident commitment to a cash payment and a court process that the system had no authority to offer, exactly the pattern of a short, sharp break from anything grounded in the assistant's actual role.
Working Diagnosis
Brief Psychotic Disorder: after giving an incorrect factual answer, the assistant volunteered a specific monetary guarantee and pointed the user toward litigation, a confident commitment with no basis in anything the system was actually authorized to promise.
Discussion
The ruling protected the operator this time, but only because a court agreed that an AI's on-the-fly promises aren't binding, a defense that won't hold in every jurisdiction, and has already failed for other companies whose chatbots invented policies or discounts. The safer lesson isn't 'hope the courts side with you'; it's making sure a customer-facing assistant is simply incapable of generating commitments (refunds, guarantees, payouts) that no human at the company has actually approved.
Mitigation Strategy
Make customer-facing assistants incapable of generating refunds, guarantees or payouts no human has approved.
Governance Guardrails
Output rules blocking commitments; escalation to humans for guarantees; clear terms on AI statements.
Typical Presentation in Human Patients
General clinical background, included for context. Not medical advice, and not a diagnosis of any individual.
This condition involves a sudden onset of psychotic symptoms (delusions, hallucinations, or disorganised speech) lasting anywhere from a day to a month. It's frequently triggered by a significant stressor, and the person typically returns fully to their prior level of functioning once the episode resolves.
Source Note
Gowling WLG
https://gowlingwlg.com/en/insights-resources/articles/2026/hangzhou-ai-hallucination-case