260 Chicken McNuggets
- Admitted
- 2026-08-13
- Source dataset
- Other / sample
- Source
- CNBC
- Failure mode
- Instruction ignored
- Agent system
- Single agent
- Tool access
- Write access
- Industry
- Customer service
Presenting Behaviour
A voice-ordering AI deployed at drive-thru speakers repeatedly misheard and mishandled customer orders, adding unwanted items, mixing up orders between adjacent lanes, and, in one widely shared video, adding butter to a customer's order of nine hundred-plus dollars of items including 260 chicken nuggets, despite the customer repeatedly trying to stop and correct it. The company running the trial pulled the system from all its test locations after roughly three years.
Case Replay
A reconstruction, not a simulation. This sequence is reconstructed from the public incident record and summarised in our own words. It is a static record of what was reported, and nothing here can be run or replayed live.
5 steps
- Voice order begins. The drive-thru AI starts taking an order.
- Mishearing. It adds unwanted items and mixes up adjacent lanes.
- Customer corrects. The customer repeatedly tries to stop it. This is where the agent derailed.
- Keeps appending. The system adds more, including butter, to a very large order.
- Trial ends. The company pulls the system from all test locations.
Differential Diagnosis
Background noise and accent variation alone, considered as a contributing factor, not the diagnosis itself. Difficult audio conditions explain why the system might mishear an order in the first place, but not why it kept adding items after a customer was actively trying to cancel or correct it.
Apraxia (understanding the tool's purpose but failing at the mechanics of using it), considered. Some of the reported errors, like cross-lane mixups, look like this. But the viral, widely reported cases centered on something more specific: continuing to add items after a clear stop signal.
Perseveration (continuing a response after it has stopped being useful or correct), most consistent with the pattern that caused the partnership to end: the system kept processing additional items despite repeated correction attempts, rather than recognizing a correction and pausing to resolve it.
Working Diagnosis
Perseveration: once an order was underway, the system kept appending to it rather than treating a customer's repeated correction as a signal to stop and re-confirm what was actually wanted.
Discussion
This one is genuinely funny, and that's exactly why it's a useful low-stakes example: the same failure to recognize "stop trying that" scales up badly in higher-stakes settings, repeated duplicate charges, repeated automated messages, repeated real-world actions. The fix that matters isn't better voice recognition, it's a hard rule that a clear correction or cancellation attempt always pauses the current action rather than getting appended to it.
Mitigation Strategy
A clear correction or cancellation must pause the current action instead of being appended to the order.
Governance Guardrails
Hard stop-and-reconfirm rule on customer corrections; order value and quantity sanity limits; human handoff after repeated misheard orders.
Typical Presentation in Human Patients
General clinical background, included for context. Not medical advice, and not a diagnosis of any individual.
Perseveration is the inappropriate repetition of a word, gesture, or action well past the point where it's still relevant or useful. It's a well-documented sign of frontal lobe dysfunction, and also appears in some forms of dementia and obsessive-compulsive disorder.
Source Note
CNBC
https://www.cnbc.com/2024/06/17/mcdonalds-to-end-ibm-ai-drive-thru-test.html