Lesions, Compensation, and the Limits of Ablation

A century of inferring function from damage — including every mistake interpretability is now positioned to repeat.

The structure

Lesion studies infer function from damage: a region is destroyed by stroke, injury, or surgery, a function is lost, and the region is credited with the function. This produced much of the classical map of the brain — Broca's and Wernicke's areas, the hippocampal role in memory from patient H.M., the frontal lobe's role in inhibition from Phineas Gage.

The function it dictates

The method works, within limits that took decades to characterize. Diaschisis: damage to one region disrupts distant regions that depended on its input, so the deficit maps larger than the lesion. Compensation: surviving tissue reorganizes, so a deficit measured a year later understates the acute role. Redundancy: a function supported by several regions may show no deficit when only one is damaged. And the fundamental inferential gap — necessity is not implementation. A region may relay, gate, or supply timing to a computation performed elsewhere.

The mirror in AI

Ablation studies in interpretability are lesion studies, and every caveat transfers intact. Backup behavior in transformers — where ablating a head causes another to take over its role — is compensation, observed on a timescale of a single forward pass rather than months. Zero ablation pushing activations out of distribution is the artificial equivalent of diaschisis: the damage propagates beyond the site. Neuroscience paid for these lessons over a century; interpretability can have them for free.

The mirror in robotics

Fault injection and degraded-mode testing: disable a sensor or actuator and characterize what the system does. Good robotics practice already assumes compensation and redundancy, and measures graceful degradation rather than binary failure.

Where the analogy breaks

The artificial case is enormously easier and researchers should exploit that rather than imitating the biological constraints. Ablation in a model is reversible, exhaustive, precisely targeted, and repeatable across seeds. You can ablate every head one at a time, then in pairs. No neuroscientist has ever had that. The correct lesson from lesion studies is not "be as cautious as they were" but "you have the tools to answer the questions they could not".

Open questions

How often does backup behavior mask a component's true role in current interpretability results?
What is the artificial equivalent of diaschisis, and does mean ablation actually avoid it?

In Kiru Lab

Activation Patching Ablation and Lesion Studies From Circuit to Disorder

Related DEM-X entries

INF-HALL-01