Sparse Coding and the Energy Budget

Spikes cost energy, so the cortex represents the world with as few active cells as it can.

The structure

Cortical activity is sparse: at any moment only a small percentage of neurons in a region are firing above baseline, and the distribution of firing rates is heavy-tailed. This is enforced by a dense network of inhibitory interneurons — several distinct types, targeting different parts of the pyramidal cells they regulate — which impose competition so that strong responses suppress their neighbors.

The function it dictates

Sparse codes are metabolically cheap, since spikes dominate the energy budget. They are also computationally useful: sparse representations are more separable, easier to read out with a simple downstream classifier, and less prone to interference between stored patterns. Olshausen and Field showed that optimizing for sparse reconstruction of natural images produces receptive fields resembling those measured in V1 — a rare case where an efficiency principle predicted the anatomy.

The mirror in AI

This is the direct ancestor of sparse autoencoders in mechanistic interpretability. The setup is nearly identical: an overcomplete dictionary, an L1 sparsity penalty, and a reconstruction objective. Where the neuroscience used it to explain measured receptive fields, interpretability uses it to disentangle superposed features in a language model. The mathematics is the same; only the system under study changed.

The mirror in robotics

Event-driven sensing and sparse state estimation follow the same logic: transmit and process only what changed. On a power-constrained mobile robot this is not an elegance argument but a battery-life one, which is precisely the pressure that shaped the cortical solution.

Where the analogy breaks

Cortical sparsity is enforced by dedicated inhibitory circuitry that costs energy to run; artificial sparsity is imposed by a penalty term in a loss function. More importantly, the brain's sparsity is a constraint it must satisfy, while a model's is a hyperparameter someone chose — which means the frequent claim that transformers "are sparse like the brain" is describing a coincidence of form, not of cause.

Open questions

Is the sparsity level in cortex optimal for computation or merely affordable?
Do sparse autoencoder features correspond to anything as principled as a V1 receptive field?

In Kiru Lab

Activations: Why the Nonlinearity Is Structural Features, Directions, and Superposition Sparse Autoencoders and Dictionary Learning