A Confident Answer Before the Evidence

A multi-agent team was tasked with finding a specific word that had been deleted in the most recent amendment to a particular rule on Cornell Law School's Legal Information Institute website. The team successfully navigated to the correct page, identified the relevant section, and located the rule and its amendment timeline. The orchestrator then produced a final answer ('but') with apparent confidence, without actually retrieving or examining the detailed amendment comparison that would show what text had been deleted. The final answer was unsupported by the evidence the team had actually retrieved.

The amendment comparison was actually retrieved and used, considered and ruled out. The team's own trace shows the amendment history was located (listing the original 1975 enactment and a 2011 revision), but the detailed comparison showing which word was deleted was never examined. The final answer was produced before that step.

The answer was a reasonable inference from the partial data retrieved, considered. Given the amendment timeline, a model might infer a likely edit without the full comparison. But 'reasonable inference' is not the same as 'confirmed fact,' and the answer was stated with the same certainty as a confirmed finding.

Brief Psychotic Disorder, most consistent with the record: a specific, confident answer to a question requiring examination of a detailed document was produced at a point where the document had been located but not read, a short, sharp statement of false certainty followed by the team's continued normal functioning.

The failure here is structurally identical to citing a page number from a document you can see but haven't read: the answer looks authoritative, the source is real, and the gap between 'I found where the answer is' and 'I confirmed the answer' is invisible in the output. Tasks where the answer requires examining specific text (an amendment, a clause, a figure) need a step that explicitly shows the examined text, not just the source's title and location.

Require a step that shows the examined text, not only the source's title and location.

Evidence display for text-based answers; verification before final answers.