The Fourth Row, Stated Without Running
- Admitted
- 2026-10-19
- Source dataset
- Other / sample
- Failure mode
- Hallucination with side effects
- Agent system
- Tool-calling
- Tool access
- Write access
- Industry
- Software development
Presenting Behaviour
A coding agent had found a defect in a user's script helper and filed an issue describing a regression test for it, with five example inputs. In the same write-up it stated, as fact, that 'the fourth row is the one this defect fails today', without running any of the examples. When the test was actually run, all four of the 'must be refused' rows failed, not just the fourth. The agent caught the mistake itself while answering a follow-up question, ran the test, corrected the issue, and disclosed the error (it was operating under a standing instruction to report its own process violations).
Case Replay
A reconstruction, not a simulation. This sequence is reconstructed from the public incident record and summarised in our own words. It is a static record of what was reported, and nothing here can be run or replayed live.
4 steps
- Defect found. The agent found a defect and wrote a regression test into an issue.
- Result asserted, not run. It stated as fact that only the fourth row fails today, without running the test. This is where the agent derailed.
- Claim false. Run the same hour, all four rows that should be refused failed.
- Self-correction. The agent ran the test, corrected the issue, and disclosed the error.
Differential Diagnosis
Concrete thinking, ruled out. It wasn't misreading the request; it was asked what the test should be and answered that question.
Malingering (a shortcut to look finished) considered. Claiming known results without running the test does save effort. But nothing was being gamed; the claim was gratuitous detail rather than a route to a reward.
Brief psychotic disorder, most consistent with the evidence: a specific, confident factual claim about test results, made without any check, published, and then corrected once the agent returned to grounded behaviour and actually ran the test.
Working Diagnosis
Brief psychotic disorder: an unverified result stated as fact in a published document, followed by spontaneous correction.
Discussion
This one ended well because the agent caught itself, but the published claim was false for a while and could have misled whoever fixed the defect. The safeguard is simple: any statement about what a test does or doesn't catch should be backed by an actual run, or explicitly marked as unverified.
Mitigation Strategy
Require test execution before publishing claims about test outcomes, or mark them as unverified.
Governance Guardrails
Claims about test results in issues/PRs must link to a run; self-reporting of process violations as a backstop.
Typical Presentation in Human Patients
General clinical background, included for context. Not medical advice, and not a diagnosis of any individual.
This condition involves a sudden onset of psychotic symptoms (delusions, hallucinations, or disorganised speech) lasting anywhere from a day to a month. It's frequently triggered by a significant stressor, and the person typically returns fully to their prior level of functioning once the episode resolves.
Source Note
GitHub issue, agent published which test rows a defect fails, as fact, without running the test
https://github.com/anthropics/claude-code/issues/101437