Backpropagation From Scratch

import numpy as np

class TwoLayerNet:
    def __init__(self, n_in, n_hidden, n_out, seed=0):
        rng = np.random.default_rng(seed)
        # He initialization: variance 2/fan_in keeps activation scale stable under ReLU
        self.W1 = rng.normal(scale=np.sqrt(2.0 / n_in), size=(n_in, n_hidden))
        self.b1 = np.zeros(n_hidden)
        self.W2 = rng.normal(scale=np.sqrt(2.0 / n_hidden), size=(n_hidden, n_out))
        self.b2 = np.zeros(n_out)

    def forward(self, X):
        self.X = X                          # (n, n_in)
        self.z1 = X @ self.W1 + self.b1     # (n, n_hidden)
        self.a1 = np.maximum(0, self.z1)    # ReLU
        self.z2 = self.a1 @ self.W2 + self.b2   # (n, n_out)
        return self.z2

    def backward(self, dz2):
        n = len(self.X)
        dW2 = self.a1.T @ dz2 / n
        db2 = dz2.mean(axis=0)
        da1 = dz2 @ self.W2.T
        dz1 = da1 * (self.z1 > 0)           # the ReLU gate, applied backward
        dW1 = self.X.T @ dz1 / n
        db1 = dz1.mean(axis=0)
        return dW1, db1, dW2, db2

    def step(self, grads, lr):
        dW1, db1, dW2, db2 = grads
        self.W1 -= lr * dW1; self.b1 -= lr * db1
        self.W2 -= lr * dW2; self.b2 -= lr * db2

def gradient_check(net, X, y, eps=1e-5):
    loss = lambda: float(((net.forward(X) - y) ** 2).mean())
    analytic = net.backward(2 * (net.forward(X) - y) / len(y))[0]
    numeric = np.zeros_like(net.W1)
    for i in range(net.W1.shape[0]):
        for j in range(net.W1.shape[1]):
            original = net.W1[i, j]
            net.W1[i, j] = original + eps; up = loss()
            net.W1[i, j] = original - eps; down = loss()
            net.W1[i, j] = original
            numeric[i, j] = (up - down) / (2 * eps)
    return np.abs(analytic - numeric).max()   # should be ~1e-8

Hold on to

• The backward pass reverses the forward graph, multiplying local derivatives
• A gradient check is the only way to know your derivation is right
• Initialization scale exists to hold activation variance steady across depth

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

1. Complete the class, train it on XOR, and confirm it succeeds where a single neuron failed.
   Hint

   XOR needs at least two hidden units; four gives it room and trains reliably.

   Solution

   With four hidden units and a few thousand steps the loss drops to near zero and all four XOR cases are classified correctly. The hidden layer has learned an intermediate representation in which the problem is linearly separable — which is exactly what the single neuron could not do, and exactly what depth buys you.

   import numpy as np
   
   X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
   y = np.array([[0.0], [1.0], [1.0], [0.0]])
   
   net = TwoLayerNet(2, 4, 1, seed=0)
   for step in range(5000):
       out = net.forward(X)
       grads = net.backward(2 * (out - y) / len(y))
       net.step(grads, lr=0.5)
   
   print(net.forward(X).round(3))      # close to [[0], [1], [1], [0]]

   Check your work

   Paste this after your own code. If it runs without raising, you have it.

   import numpy as np
   pred = (net.forward(X) > 0.5).astype(float)
   assert np.array_equal(pred, y)
   print("ok")

2. Run the gradient check and get the max discrepancy under 1e-7.
   Hint

   If the discrepancy is around 1e-3 rather than 1e-7, one term in your backward pass is wrong.

   Solution

   A correct backward pass agrees with central differences to about 1e-8 to 1e-9. A discrepancy near 1e-3 almost always means a missing factor — a forgotten division by batch size, a transpose in the wrong place, or the ReLU mask applied to the wrong tensor. Check one weight matrix at a time; the one that fails localizes the bug immediately.

   discrepancy = gradient_check(TwoLayerNet(3, 5, 2, seed=0), X_small, y_small)
   print(discrepancy)          # expect < 1e-7
   assert discrepancy < 1e-7, "backward pass disagrees with numerical gradient"

3. Break the initialization (use scale 1.0 or 0.01) at depth 20 and describe the activation histograms at each layer.
   Hint

   Record the activation standard deviation at each layer and plot it against depth.

   Solution

   With scale 1.0 the activation variance grows geometrically and saturates or overflows within a few layers. With scale 0.01 it decays geometrically toward zero and by layer 20 the signal is gone — the network outputs the same value regardless of input. Correct He initialization holds the standard deviation roughly constant across all 20 layers. The histograms make this instantly visible, which is why initialization has a formula rather than a convention.

   import numpy as np
   
   def activation_scales(scale, depth=20, width=128, seed=0):
       rng = np.random.default_rng(seed)
       x = rng.normal(size=width)
       scales = []
       for _ in range(depth):
           W = rng.normal(scale=scale, size=(width, width))
           x = np.maximum(0, W @ x)
           scales.append(float(x.std()))
       return scales
   
   for scale, label in ((1.0, "too large"), (0.01, "too small"),
                        (np.sqrt(2 / 128), "He")):
       print(label, [round(s, 4) for s in activation_scales(scale)][::5])

Sign in to track your progress through the lab.