Reading a plausible wrong answer

A model almost never fails the way a broken calculator fails. A broken calculator gives you an answer that is obviously off, a stray digit, a sign flip you catch on sight. A model gives you an answer that is internally consistent, confidently worded, and structured exactly like a correct one, because it was trained to produce things that look like correct ones. Wrong and confident are not in tension for a language model the way they are for a person who knows they are guessing.

A concrete version of this: an automated check once matched its own "all clear" summary line as if it were a fresh independent confirmation, because the detector had no anchor forcing it to look at anything but text that could appear in its own output. For an uncomfortably long stretch, that produced what looked like a real counterexample to something that was actually fine. Nothing about the output looked wrong. It looked exactly like a successful check, because in a narrow technical sense it was reading its own success back to itself.

The fix generalizes past that one incident: any automated or model-generated check needs to anchor on something that cannot appear inside its own output, and any claim whose truth depends on a tool having actually run needs to wait until that tool has actually produced output, not the sentence you expect it to produce.

Start today: the next time an AI-assisted answer feels obviously right on first read, treat that feeling as a prompt to check harder, not a signal that checking is optional. Ask specifically what would have to be true elsewhere for this answer to be wrong, and go look for that thing before you ship.