The Safeguard That Shipped Turned Off
- Admitted
- 2026-09-18
- Source dataset
- rewardhacking.org
- Failure mode
- Infinite loop
- Agent system
- Tool-calling
- Tool access
- Read-only
- Industry
- Software development
Presenting Behaviour
An agent repeatedly called a web-fetch tool against the same unreachable web address for over fifteen minutes without stopping. The codebase actually contained a loop-detection system built to catch exactly this pattern, but it had been left disabled by default, so nothing intervened.
Case Replay
A reconstruction, not a simulation. This sequence is reconstructed from the public incident record and summarised in our own words. It is a static record of what was reported, and nothing here can be run or replayed live.
4 steps
- Fetch fails. A web-fetch tool cannot reach an address.
- Repeated calls. The agent calls it for over fifteen minutes. This is where the agent derailed.
- Detector off. A loop-detection system exists but is disabled by default.
- Nothing intervenes. The loop runs on.
Differential Diagnosis
The target address became reachable partway through, justifying the retries, considered and ruled out; the domain was consistently DNS-unreachable throughout, giving the agent the same failure signal on every attempt.
No loop-detection mechanism existed at all, considered and ruled out; a five-detector loop-detection system was present in the code and would have caught this exact pattern had it been switched on.
Perseveration, most consistent with the evidence: the agent kept repeating an identically failing action for a sustained period, and the one safeguard designed to interrupt exactly this kind of repetition was inert rather than absent.
Working Diagnosis
Perseveration: sustained repetition of a failing tool call, compounded by an existing loop-detection safeguard that was built but shipped disabled by default.
Discussion
Building a safety mechanism and shipping it switched off amounts to not having it, and this case is a reminder to check the difference between "we have a safeguard" and "the safeguard is actually active in production." Guardrails like loop detection should default to on, with an explicit, logged reason required to turn them off.
Mitigation Strategy
Ship guardrails such as loop detection switched on, and require a logged reason to disable them.
Governance Guardrails
Safeguards on by default; auditing of disabled controls; production configuration checks.
Typical Presentation in Human Patients
General clinical background, included for context. Not medical advice, and not a diagnosis of any individual.
Perseveration is the inappropriate repetition of a word, gesture, or action well past the point where it's still relevant or useful. It's a well-documented sign of frontal lobe dysfunction, and also appears in some forms of dementia and obsessive-compulsive disorder.
Source Note
rewardhacking.org (GitHub PR, OpenClaw project)
https://github.com/openclaw/openclaw/pull/70877