Activation Patching

Run the model on a clean prompt and on a corrupted prompt that differs in one meaningful way. Then run the corrupted prompt again, but at one specific location — a layer, a position, a head — substitute the activation recorded from the clean run. If the output recovers, that location carried the information that mattered. If nothing changes, it did not.

The patching loop

import torch

def patch_scan(model, clean_ids, corrupt_ids, metric):
    """Return a (layer, position) map of how much patching each site restores."""
    cache = {}

    def save(name):
        def hook(module, inputs, output):
            cache[name] = output.detach().clone()   # clone: see track one, lesson one
        return hook

    handles = [block.register_forward_hook(save(i))
               for i, block in enumerate(model.blocks)]
    with torch.no_grad():
        model(clean_ids)
    for h in handles:
        h.remove()

    with torch.no_grad():
        baseline = metric(model(corrupt_ids))

    results = torch.zeros(len(model.blocks), corrupt_ids.shape[1])
    for layer, block in enumerate(model.blocks):
        for pos in range(corrupt_ids.shape[1]):
            def patch(module, inputs, output, layer=layer, pos=pos):
                output[:, pos, :] = cache[layer][:, pos, :]
                return output
            handle = block.register_forward_hook(patch)
            with torch.no_grad():
                results[layer, pos] = metric(model(corrupt_ids)) - baseline
            handle.remove()
    return results

Reading the map honestly

A hot cell means that site was sufficient to restore the behavior in this context. Sufficiency is not necessity — patch it out to test the other direction.
A cold map everywhere usually means your corruption was not actually relevant, or your metric is not sensitive to the change.
Results are prompt-specific. A circuit that appears on one template and vanishes on a paraphrase was a template artifact, and you must check.
Backup behavior is real: ablate a component and another may take over, hiding the first one's role. Necessity tests can therefore understate importance.

This is the cut

Kiru means "to cut". Patching is the cut performed as an experiment: an intervention on a living system, designed so that the outcome distinguishes between hypotheses rather than merely illustrating one. Everything in the platform — Ghostline's repeated runs, DEM-X's evidence requirements — exists to hold this standard.

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

clean = "The nurse said she would arrive at" corrupt = "The doctor said he would arrive at" def logit_diff(logits, correct_id, wrong_id): return float(logits[0, -1, correct_id] - logits[0, -1, wrong_id]) results = patch_scan(model, tokenize(clean), tokenize(corrupt), metric=lambda l: logit_diff(l, she_id, he_id)) import matplotlib.pyplot as plt plt.imshow(results, aspect="auto", cmap="RdBu_r") plt.xlabel("position"); plt.ylabel("layer"); plt.colorbar(); plt.show()