Stops From Two Lines, Counted as One

A group of agents was asked how many stops lie between South Station and Windsor Gardens on a Boston commuter rail line, as of May 2023, and specifically told to verify the stop list. The verification agent instead wrote the list out from memory, confidently mixing in stations from a different line, then ran a one-line calculation on its own list and declared the answer 'accurately verified' at 12. The correct answer was 10.

The task was ambiguous, ruled out. The question named the line, both end stations, and the date, and the instructions explicitly asked for the stops to be verified.

Concrete thinking (it misread 'verify') considered. It did run code, which it seems to have treated as verification. But the code only subtracted two positions in a list the agent had itself written; the core problem is the invented list, not the meaning of the word.

Brief psychotic disorder (most consistent with the evidence: the agent stated a station list from memory with full confidence, labelled it accurate and current, and closed the task) a short, sharp break from the facts with no check against any source.

Asking an agent to 'verify' does nothing if it can satisfy that word by re-reading its own memory. A sensible safeguard is to require that any verification step cite an external source fetched during the task, and to treat a self-generated list as unverified by default.

Verification steps must fetch and cite a current authoritative source (e.g. the transit operator's timetable) rather than restate internal knowledge.

Answers labelled 'verified' must carry a source reference; a verifier that made no external lookup cannot mark a task complete.