Bio Mirror / Failure, Narrative, and the Limits of Analogy

Confabulation and the Interpreter

Humans produce fluent, sincere, false accounts to fill gaps. So do models. The behaviors rhyme; the mechanisms do not match.

The structure

Confabulation appears after damage to ventromedial prefrontal and basal forebrain regions, notably in Korsakoff syndrome. Patients produce detailed, coherent accounts of events that did not happen, without any intent to deceive and typically without doubt. Gazzaniga's split-brain experiments produced a cleaner demonstration: when the right hemisphere was shown an instruction and acted on it, the verbal left hemisphere — with no access to the instruction — immediately generated a confident explanation for the action, which it believed. Gazzaniga called this the interpreter.

The function it dictates

The system that generates explanations is not the system that performs actions, and it does not have privileged access to its causes. It builds a plausible account from available evidence. When the true cause is unavailable — through damage, through disconnection, or simply because the process was not accessible to introspection — the account is constructed anyway, and it feels exactly like a memory or a reason. This is the normal operation of the mechanism, not a malfunction of it.

The mirror in AI

A model asked something outside its knowledge produces the most plausible continuation, fluently and without hedging. A model asked to explain its own reasoning produces a plausible explanation that may have no relationship to the computation that actually produced the output — chain-of-thought text is generated by the same next-token process as everything else, and there is no mechanism guaranteeing it describes the underlying computation. The parallel to the interpreter is close enough to be genuinely useful as a warning.

The mirror in robotics

A post-hoc explanation module that narrates a controller's decisions from its inputs and outputs rather than from its internal state. Such systems produce satisfying explanations that are not causal accounts, and the failure is instructive precisely because it is so easy to build accidentally.

Where the analogy breaks

State this clearly, because it is the entry most likely to be over-read. Human confabulation involves damaged memory systems, intact linguistic systems, and a self-model that the explanation must remain consistent with. A language model has no memory system in that sense, no self-model that persists between sessions, and no distinction between the system that acts and the system that explains — it is one process throughout. The behaviors converge; the mechanisms do not. Use the human case to generate predictions about when models will confabulate, then test those predictions on the model with the causal methods from the interpretability track. Never cite the human case as evidence about the machine.

Open questions

Is chain-of-thought text ever a faithful description of the computation that produced an answer?
Does the interpreter analogy predict anything about models that a purely mechanical account does not?

In Kiru Lab

Why Models Confabulate Probing: What a Classifier on Activations Can and Cannot Tell You From Circuit to Disorder

Related DEM-X entries

INF-HALL-01