Kiru Lab  /  Deep Learning  /  Training Dynamics and Failure Modes

Reading a Failing Run

A diagnostic table for training failures — the same instinct you will apply to model behavior in DEM-X.

Hands-On Lab  ·  about 35 minutes

Training failures have signatures. Learning to read them is a diagnostic skill, and it is the same skill DEM-X asks for: observe a symptom, generate hypotheses, run the test that distinguishes them.

  • Loss is NaN: learning rate too high, unscaled inputs, or a log/divide of zero. Bisect by lowering the rate 10x and checking input statistics.
  • Loss flat from step one: gradients are not reaching parameters. Check for a detached graph, a frozen module, or a zero learning rate.
  • Training loss falls, validation rises: overfitting. Add data, add regularization, or reduce capacity.
  • Both losses plateau high: underfitting, or an unlearnable target. Test by overfitting a batch of ten examples deliberately — if you cannot drive that to zero, the problem is the setup, not the data.
  • Loss spikes periodically: a bad batch or a data-loader ordering artifact. Log the batch index at each spike.
  • Validation better than training: usually dropout or augmentation being active only during training. Not always a bug.
Instrument first, guess second
def gradient_report(model):
    for name, param in model.named_parameters():
        if param.grad is None:
            print(f"{name:40s} NO GRADIENT")   # detached or frozen
            continue
        g = param.grad
        print(f"{name:40s} norm={g.norm():.3e}  max={g.abs().max():.3e}  zeros={(g == 0).float().mean():.2%}")
The overfit-ten-examples test

Before debugging anything else, confirm your model can drive the loss to near zero on ten examples. If it cannot, no hyperparameter will save you — there is a wiring bug. This single test resolves more training problems than any other, and it costs a minute.

Hold on to

  • Training failures have readable signatures
  • Overfitting ten examples is the fastest wiring check available
  • Instrument gradients before changing hyperparameters

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Deliberately induce four of the six failures above and record the curve for each.
    Hint

    Induce each one deliberately by breaking exactly one thing.

    Solution

    NaN: set the learning rate to 100. Flat from step one: call `.detach()` on the loss before backward, or set requires_grad False. Train down, validation up: train a large model on 50 examples for many epochs. Both plateau high: shuffle the labels so there is nothing learnable. Each has an unmistakable curve shape once you have seen it once, which is the whole reason to induce them on purpose rather than meeting them at 2am.

  2. Add `gradient_report` to a training loop and identify which layer has the largest gradient norm at initialization.
    Hint

    Call it right after `loss.backward()` and before the optimizer step.

    Solution

    At initialization the largest gradient norms are usually in the final layers, because they are closest to the loss and the signal has not yet been attenuated by depth. If an early layer shows a norm many orders of magnitude smaller, you are watching the vanishing-gradient product directly. A `None` gradient means that parameter is disconnected from the loss — frozen, detached, or simply unused.

    loss = criterion(model(x), y)
model.zero_grad()
loss.backward()
gradient_report(model)          # before optimizer.step()

Related DEM-X entries

GI-DRFT-01

Sign in to track your progress through the lab.

← Optimizers, Schedules, and Normalization Back to track →