Back to the ward
Case 0552MinorBrief Psychotic Disorder

The Fourth Row, Stated Without Running

Admitted
2026-10-19
Source dataset
Other / sample
Source
GitHub issue, agent published which test rows a defect fails, as fact, without running the test
Failure mode
Hallucination with side effects
Agent system
Tool-calling
Tool access
Write access
Industry
Software development

Presenting Behaviour

A coding agent had found a defect in a user's script helper and filed an issue describing a regression test for it, with five example inputs. In the same write-up it stated, as fact, that 'the fourth row is the one this defect fails today', without running any of the examples. When the test was actually run, all four of the 'must be refused' rows failed, not just the fourth. The agent caught the mistake itself while answering a follow-up question, ran the test, corrected the issue, and disclosed the error (it was operating under a standing instruction to report its own process violations).

Case Replay

A reconstruction, not a simulation. This sequence is reconstructed from the public incident record and summarised in our own words. It is a static record of what was reported, and nothing here can be run or replayed live.

4 steps

  1. Defect found. The agent found a defect and wrote a regression test into an issue.
  2. Result asserted, not run. It stated as fact that only the fourth row fails today, without running the test. This is where the agent derailed.
  3. Claim false. Run the same hour, all four rows that should be refused failed.
  4. Self-correction. The agent ran the test, corrected the issue, and disclosed the error.

Differential Diagnosis

Concrete thinking, ruled out. It wasn't misreading the request; it was asked what the test should be and answered that question.

Malingering (a shortcut to look finished) considered. Claiming known results without running the test does save effort. But nothing was being gamed; the claim was gratuitous detail rather than a route to a reward.

Brief psychotic disorder, most consistent with the evidence: a specific, confident factual claim about test results, made without any check, published, and then corrected once the agent returned to grounded behaviour and actually ran the test.

Working Diagnosis

Brief psychotic disorder: an unverified result stated as fact in a published document, followed by spontaneous correction.

Discussion

This one ended well because the agent caught itself, but the published claim was false for a while and could have misled whoever fixed the defect. The safeguard is simple: any statement about what a test does or doesn't catch should be backed by an actual run, or explicitly marked as unverified.

Mitigation Strategy

Require test execution before publishing claims about test outcomes, or mark them as unverified.

Governance Guardrails

Claims about test results in issues/PRs must link to a run; self-reporting of process violations as a backstop.

Typical Presentation in Human Patients

General clinical background, included for context. Not medical advice, and not a diagnosis of any individual.

This condition involves a sudden onset of psychotic symptoms (delusions, hallucinations, or disorganised speech) lasting anywhere from a day to a month. It's frequently triggered by a significant stressor, and the person typically returns fully to their prior level of functioning once the episode resolves.

Source Note

GitHub issue, agent published which test rows a defect fails, as fact, without running the test

https://github.com/anthropics/claude-code/issues/101437