Kiru Lab  /  Mechanistic Interpretability  /  From Mechanism to Diagnosis

From Circuit to Disorder

The closing lesson. Everything in this lab exists to make this document possible.

Bridge  ·  about 45 minutes

DEM-X catalogs AI failures as disorders: named, reproducible behavioral conditions with diagnostic criteria. A behavioral description is where an entry starts. A mechanistic account is what makes it diagnostic — it turns "the model does X" into "the model does X because of Y, which predicts it will also do Z under condition W".

The structure of a strong entry

  1. Behavioral description: what the model does, stated so a stranger could recognize it.
  2. Reproduction: exact prompts, model version, decoding configuration, seeds, and observed rate across runs — not one screenshot.
  3. Mechanistic hypothesis: which architectural or training property you believe produces it.
  4. Causal evidence: the patching, ablation, or steering result that distinguishes your hypothesis from the obvious alternative.
  5. Predictions: what else must be true if you are right, including at least one that could fail.
  6. Boundaries: which models, scales, and conditions you tested, and where you did not look.
  7. Human parallel, if any: stated as analogy, with the mechanistic differences named explicitly.
The standard

A claim that cannot fail is not a finding. If no observation you can describe would change your mind about a mechanistic account, you have written a story about the model rather than a description of it. Write down, before you run the experiment, what result would make you abandon the hypothesis.

What you can do now

You started this curriculum with names and objects in Python. You can now read a model's architecture and predict which failures it structurally permits, design an intervention that tests a mechanistic claim, and report a result at a standard that lets someone else check it. That is the working definition of an AI-surgery practitioner, and it is what the rest of Kiru is built for.

The Bio Mirror section runs alongside this one for a reason. Neuroscience has spent a century on the inverse problem you are now working on — inferring computation from structure, with lesions, recordings, and interventions as the only tools. Its methods are worth borrowing, and so are its cautionary tales. Read it as a source of hypotheses and of discipline, never as evidence about a system made of matrices.

Hold on to

  • A mechanistic account turns a description into a diagnosis
  • A claim that cannot fail is not a finding
  • Write down the falsifying result before running the experiment

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Take a failure you have personally observed and write a complete DEM-X entry against the seven-part structure above.
    Hint

    Use the seven headings from the lesson as literal section headings in your draft.

    Solution

    The section people skip is boundaries — which models, scales, and conditions you tested and where you did not look — and it is the one that makes an entry trustworthy. An entry claiming a general property from one model at one scale is weaker than one claiming a narrow property with its limits stated, even though the first sounds more impressive. Write the boundaries section first if you find yourself overclaiming.

  2. Have someone else attempt your reproduction steps on a different machine. Record every step that was ambiguous and fix it.
    Hint

    Watch them work through it without helping, and write down every question they ask.

    Solution

    The ambiguities are almost always unstated assumptions: which model version, which decoding settings, which system prompt, what counts as the behavior occurring, and how many runs. Every question they had to ask is a defect in the writeup. Fix each one in the text rather than answering it in conversation — the next reader will not have you available.

  3. Write the falsification criterion for your mechanistic hypothesis, then run the experiment that could meet it.
    Hint

    Write the sentence "I would abandon this hypothesis if ___" before running anything.

    Solution

    A usable falsification criterion is specific and measurable: "if ablating these three heads leaves the behavior rate within 5% of baseline across 200 prompts and three seeds, the circuit hypothesis is wrong." Then run it. If you cannot complete that sentence, you do not yet have a mechanistic hypothesis — you have a description with mechanistic vocabulary attached, and the difference is the whole subject of this track.

In the Bio Mirror

Lesions, Compensation, and the Limits of Ablation Confabulation and the Interpreter

Related DEM-X entries

INF-HALL-01 GI-SYCO-01 GI-DRFT-01

Sign in to track your progress through the lab.

← Sparse Autoencoders and Dictionary Learning Back to track →