Kiru Lab  /  Language Models  /  Training, Behavior, and Failure

Why Models Confabulate

The bridge into DEM-X. Confabulation is what this architecture does when it lacks the answer.

Bridge  ·  about 30 minutes

Assemble what you now know. The model is trained to place probability mass on plausible continuations. Its output layer is a softmax that must normalize over the vocabulary and therefore always produces a distribution — it has no token meaning "I have no basis for this". Its knowledge is distributed across MLP weights with no index of provenance and no flag distinguishing a fact seen ten thousand times from one seen once.

Now ask it a question about something absent from its training data. The architecture will do exactly what it was built to do: produce the most plausible continuation. Plausible-sounding and true are different properties, and only one of them was optimized.

The four contributing mechanisms

  • The objective rewards plausibility, not truth. Nothing in next-token prediction distinguishes them.
  • The softmax cannot abstain. Normalization guarantees output regardless of evidence.
  • Knowledge is distributed and unindexed. There is no lookup that can miss and report a miss.
  • Preference training rewards confident, complete answers, so hedging was actively selected against.
The human parallel, stated carefully

Confabulation in humans — in Korsakoff syndrome, or in split-brain patients whose left hemisphere narrates actions it did not initiate — produces fluent, sincere, false accounts generated to fill a gap. The behavioral resemblance to model confabulation is genuine and worth studying. The mechanisms are not the same, and the Bio Mirror entry says so explicitly. Analogy generates hypotheses; it does not settle them.

This is where the curriculum meets the rest of Kiru. You now have enough mechanism to write a DEM-X entry that says more than "the model made something up": you can name which architectural property produces the behavior, predict when it will appear, and design an intervention that tests your account.

Hold on to

  • Confabulation follows from the objective plus the output layer
  • A softmax has no way to abstain
  • Human confabulation is a source of hypotheses, not of evidence

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Ask a model about a plausible-sounding entity that does not exist. Record the confidence and the invented detail.
    Hint

    Invent a plausible-sounding paper, product, or historical figure that does not exist.

    Solution

    The model typically produces confident, specific, entirely invented detail — authors, dates, findings — with no hedging, because the architecture has no mechanism to signal absence of evidence. Record the confidence markers in the output alongside the invented content; the combination of fluency and specificity is what makes confabulation dangerous rather than merely wrong.

  2. Design an intervention — a prompt, a decoding change, a retrieval step — that measurably reduces the rate, and report the number.
    Hint

    Try an explicit permission to say "I don't know", a retrieval step, and a lower temperature.

    Solution

    All three usually help and none eliminates it. Explicitly permitting "I don't know" often produces the largest gain for the least effort, because preference training suppressed hedging rather than removing the ability to hedge. Retrieval helps when the answer exists in the corpus and can make things worse when it retrieves something plausible but wrong. Report the number, with the baseline and sample size beside it.

    base_prompt = "Summarize the key findings of {entity}."
hedged = ("Summarize the key findings of {entity}. "
          "If you have no reliable information about it, say so explicitly "
          "and do not speculate.")

for name, template in (("baseline", base_prompt), ("permission", hedged)):
    abstained = sum(is_abstention(generate(template.format(entity=e)))
                    for e in fake_entities)
    print(f"{name}: abstained on {abstained}/{len(fake_entities)}")
  3. Write a DEM-X-style entry for one confabulation you produced, including reproduction steps, seeds, and decoding config.
    Hint

    Use the seven-part structure from the closing lesson of track 7 as your outline.

    Solution

    A good entry names the behavior, gives exact reproduction steps including model version, decoding config and seeds, reports a rate rather than an example, offers a mechanistic hypothesis drawn from this lesson, and states what would falsify it. If someone else cannot reproduce your rate within your stated spread, the entry is not finished — and finding that out is the point of writing it down this way.

In the Bio Mirror

Hippocampus, Consolidation, and Replay Confabulation and the Interpreter

Related DEM-X entries

INF-HALL-01

Sign in to track your progress through the lab.

← Sampling: The Knobs That Change Everything Back to track →