Where Loss Functions Come From

import numpy as np

def cross_entropy(logits: np.ndarray, target_index: int) -> float:
    shifted = logits - logits.max()                # stability, not mathematics
    probs = np.exp(shifted) / np.exp(shifted).sum()
    return float(-np.log(probs[target_index]))

# Confident and right -> near 0.  Confident and wrong -> very large.
print(cross_entropy(np.array([5.0, 0.0, 0.0]), 0))   # ~0.01
print(cross_entropy(np.array([5.0, 0.0, 0.0]), 1))   # ~5.02

Hold on to

• Every standard loss is a negative log-likelihood under some assumption
• Cross-entropy punishes confident wrongness harshly and asymmetrically
• Accuracy and calibration are independent properties

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

1. Derive squared error from a Gaussian likelihood on paper, then confirm the minimizer is the mean.
   Hint

   Write the Gaussian density, take the log, and drop the terms that do not depend on the parameter.

   Solution

   The log of a Gaussian likelihood is a constant minus the squared residual over 2-sigma-squared, so maximizing likelihood is minimizing squared error. Setting the derivative to zero gives the sample mean as the minimizer. That is why MSE regression predicts a conditional mean — and why it produces an invalid average when the target is genuinely bimodal, which is exactly the trap in the inverse kinematics lesson in track six.

   # log p(y | mu) for Gaussian noise:
   #   log p = -(y - mu)^2 / (2 sigma^2) - log(sigma sqrt(2 pi))
   # Dropping constants, maximizing log p == minimizing (y - mu)^2.
   #
   # d/dmu sum_i (y_i - mu)^2 = -2 sum_i (y_i - mu) = 0
   #   =>  mu = mean(y)

   Check your work

   Paste this after your own code. If it runs without raising, you have it.

   import numpy as np
   y = np.array([1.0, 4.0, 7.0, 2.0])
   grid = np.linspace(y.min(), y.max(), 10001)
   best = grid[np.argmin([((y - m) ** 2).sum() for m in grid])]
   assert abs(best - y.mean()) < 1e-2
   print("ok")

2. Build a reliability diagram for any classifier you have and state whether it is over- or under-confident.
   Hint

   Bin predictions by confidence, then compare each bin's mean confidence to its accuracy.

   Solution

   Modern neural networks are almost always overconfident: the curve sits below the diagonal, meaning predictions made at 90% confidence are right rather less than 90% of the time. Report expected calibration error — the weighted average gap across bins — as a single number alongside the diagram.

   import numpy as np
   
   def reliability(confidences, correct, bins=10):
       edges = np.linspace(0, 1, bins + 1)
       rows = []
       for lo, hi in zip(edges[:-1], edges[1:]):
           mask = (confidences > lo) & (confidences <= hi)
           if mask.sum() == 0:
               continue
           rows.append((confidences[mask].mean(), correct[mask].mean(), int(mask.sum())))
       ece = sum(abs(c - a) * n for c, a, n in rows) / len(confidences)
       return rows, ece

   Check your work

   Paste this after your own code. If it runs without raising, you have it.

   import numpy as np
   conf = np.array([0.95] * 100)
   correct = np.array([1] * 70 + [0] * 30)
   rows, ece = reliability(conf, correct)
   assert abs(ece - 0.25) < 0.01      # 95% confident, 70% right -> overconfident
   print("ok")

Sign in to track your progress through the lab.