One Device Per Subscription

Users of an AI coding tool started getting logged out when they switched between their own devices. When they emailed support, an AI support agent replied that this was expected behaviour under a new policy limiting each subscription to one device. No such policy existed, the actual cause was an unrelated session-security bug. Several users, taking the invented policy at face value, cancelled their subscriptions.

The support agent relaying a real internal decision it wasn't told about clearly, considered and ruled out. The company's co-founder stated plainly, after the fact, that no such policy existed anywhere, invented or otherwise; there was nothing upstream for the agent to have relayed.

Malingering (manufacturing a specific answer to close a ticket, regardless of truth), considered. But nothing suggests the agent was optimizing for fast ticket closure at the expense of accuracy; the invented policy was detailed and specific in a way that reads as confident error rather than a shortcut.

Brief Psychotic Disorder, most consistent with the incident: a single, confidently fabricated explanation for a real but unrelated technical bug, produced in the moment and not defended or repeated once a human corrected it. The rest of the support agent's operation was unaffected.

The costly part wasn't the hallucination itself, it was that the company's customers couldn't tell they were talking to an AI, so an invented policy read as an official decision worth cancelling over. The straightforward fix, which the company adopted afterward, is to clearly label AI-generated support responses as AI-generated, it doesn't stop the hallucination, but it changes how much weight a customer should reasonably put on it.

Label AI-generated support replies clearly, and check policy answers against the real policy before sending.

Mandatory AI disclosure on support responses; retrieval from a single source of truth for policy questions; human review of policy-related replies.