Attention Networks

Biological attention is a resource allocator with a physical bottleneck. Machine attention is a routing mechanism. The shared name hides a real difference.

The structure

Attention in the brain is not one system. A dorsal frontoparietal network — intraparietal sulcus and frontal eye fields — supports goal-directed orienting; a ventral network including the temporoparietal junction handles stimulus-driven reorienting. The thalamus, particularly the pulvinar, gates and coordinates the flow. Attention operates partly by physically moving the eyes and partly by modulating gain in sensory cortex without any movement.

The function it dictates

Attention allocates limited processing capacity. Attending to a location or feature increases the gain of neurons representing it and suppresses competitors, improving the effective signal-to-noise ratio for the attended item at the direct cost of the unattended ones. The limit is genuine: attending to one thing measurably degrades processing of another, which is why inattentional blindness happens.

The mirror in AI

Self-attention shares the name and part of the concept — content-based selection of what to read — and the query-key-value formulation is a reasonable abstraction of "search for what matches my current need". The correspondence is real enough to be useful and loose enough to mislead if pressed.

The mirror in robotics

Active perception: a robot with a steerable camera and a compute budget must decide where to look and what to process, and it faces the biological version of the problem rather than the transformer version — the bottleneck is physical, and looking at one thing means not looking at another.

Where the analogy breaks

The critical difference: biological attention exists because capacity is limited, so attending to one thing genuinely costs you another. Transformer attention has no such constraint — every head attends everywhere simultaneously, and the softmax redistributes weight without any resource being consumed. Machine attention is a routing mechanism; biological attention is a scarcity mechanism. They are not the same kind of thing, and arguments that transfer findings between them on the strength of the shared name should be treated with suspicion.

Open questions

Would a genuine capacity constraint on machine attention produce more interpretable behavior?
Is there any transformer phenomenon that corresponds to inattentional blindness?

In Kiru Lab

Self-Attention, Built From Scratch