Kiru Lab / Deep Learning / From Neuron to Network
Activations: Why the Nonlinearity Is Structural
Without a nonlinearity, depth is decoration. With the wrong one, depth is untrainable.
Concept · about 25 minutes
You proved this in track two: a stack of linear layers collapses to a single matrix. The nonlinearity between layers is what prevents the collapse, and it is therefore the load-bearing element of depth — not the extra parameters.
- Sigmoid: bounded to (0,1), interpretable as a probability, but saturates at both ends where its derivative approaches zero. This is what stalled deep networks before 2010.
- Tanh: zero-centered, which helps optimization, but saturates the same way.
- ReLU: max(0, x). Cheap, no saturation for positive inputs, and it made deep training practical. Its cost is that a unit pushed permanently negative has zero gradient forever — a "dead" unit.
- GELU / SiLU: smooth near zero, which behaves better in transformers and avoids the hard dead-unit boundary. These are what modern LLMs use.
import numpy as np
relu = lambda z: np.maximum(0, z)
relu_grad = lambda z: (z > 0).astype(float)
z = np.array([-3.0, -0.1, 0.0, 0.1, 3.0])
print(relu(z)) # [0. 0. 0. 0.1 3. ]
print(relu_grad(z)) # [0. 0. 0. 1. 1. ] <- zero gradient means no correction, everReLU produces genuinely sparse activations — most units output exactly zero for any given input. Cortical activity is also sparse, for energy reasons. Whether these two sparsities serve the same computational purpose is an open question, and a good one to argue about in the Bio Mirror.
Hold on to
- The nonlinearity, not the parameter count, is what makes depth meaningful
- Saturation kills gradients; ReLU trades that for dead units
- Modern LLMs use smooth variants for better transformer dynamics
Work through
Try each one before opening the solution. Getting it wrong first is most of where the learning happens.
-
Train the same small network with sigmoid and with ReLU at depth 10 and compare convergence.
Hint
Keep everything else identical — same data, same seed, same learning rate.
Solution
At depth 10 the sigmoid network barely trains: its local derivative is at most 0.25, so ten layers multiply the gradient by at most 1e-6 and the early layers receive essentially nothing. The ReLU network converges normally, because its derivative is exactly 1 wherever the unit is active. This single comparison is most of why deep learning became practical around 2010.
import numpy as np def make_net(depth, width, activation, seed=0): rng = np.random.default_rng(seed) return [rng.normal(scale=np.sqrt(2 / width), size=(width, width)) for _ in range(depth)] sigmoid = lambda z: 1 / (1 + np.exp(-z)) relu = lambda z: np.maximum(0, z) for name, act in (("sigmoid", sigmoid), ("relu", relu)): x = np.random.default_rng(0).normal(size=64) for W in make_net(10, 64, act): x = act(W @ x) print(name, "activation scale after 10 layers:", float(np.abs(x).mean())) -
Instrument a ReLU network to count dead units over training. Report the fraction at the end.
Hint
A unit is dead if it outputs zero for every input in your dataset.
Solution
Typically 5–20% of ReLU units end up dead, and the fraction rises sharply with an aggressive learning rate — a large step can push a unit's pre-activation permanently negative, after which its gradient is exactly zero forever and it can never recover. Leaky ReLU and GELU exist to keep a small gradient alive on the negative side for precisely this reason.
import numpy as np def dead_fraction(pre_activations): """pre_activations: (n_samples, n_units) before the ReLU.""" ever_active = (pre_activations > 0).any(axis=0) return float(1.0 - ever_active.mean()) # Collect pre-activations over the whole dataset, then: # print(f"dead units: {dead_fraction(collected):.1%}")Check your work
Paste this after your own code. If it runs without raising, you have it.
import numpy as np pre = np.array([[-1.0, 2.0, -3.0], [-2.0, 1.0, -1.0]]) assert abs(dead_fraction(pre) - 2/3) < 1e-9 print("ok")
Sign in to track your progress through the lab.