Sequences, Memory, and the Bottleneck That Broke Recurrence

A recurrent network processes a sequence one element at a time, carrying a hidden state forward. The state is the model's entire memory of everything before now, compressed into a fixed-size vector.

h_t = phi(W_h h_{t-1} + W_x x_t + b)

Two problems follow directly from that recurrence. First, information from step 1 must survive being multiplied by `W_h` at every subsequent step — the vanishing-gradient product from track two, now operating over sequence length instead of depth. LSTMs and GRUs mitigate it with gates that create a path where the multiplier is near one. Second, the hidden state is a fixed-size bottleneck: no matter how long the sequence, the past must compress into the same number of dimensions.

The move that resolved it

Attention removes the bottleneck by keeping every previous element available and letting the model retrieve whichever ones are relevant, rather than forcing everything through a single running summary. The cost is that comparing every position to every other is quadratic in sequence length — which is why context windows have a price and why long-context research is an active field.

Forward reference

This is the exact motivation for the next track. Attention is not a clever trick added to sequence models; it is the structural answer to a specific bottleneck you can now name precisely.

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

import numpy as np, torch, torch.nn as nn def copy_batch(length, batch=64, vocab=8): x = torch.randint(1, vocab, (batch, length)) return x, x.clone() for length in (10, 50, 200): model = nn.RNN(input_size=16, hidden_size=64, batch_first=True) # train briefly, then evaluate exact-match accuracy on held-out sequences print(length, evaluate(model, length))