Overfitting and Capacity

import numpy as np
rng = np.random.default_rng(0)
x = np.linspace(0, 1, 12)
y = np.sin(2 * np.pi * x) + rng.normal(scale=0.15, size=x.shape)

for degree in (1, 3, 11):
    coeffs = np.polyfit(x, y, degree)
    train_err = np.mean((np.polyval(coeffs, x) - y) ** 2)
    xt = np.linspace(0, 1, 200)
    test_err = np.mean((np.polyval(coeffs, xt) - np.sin(2 * np.pi * xt)) ** 2)
    print(f"degree {degree:2d}  train {train_err:.4f}  true-function {test_err:.4f}")
# degree 11 has near-zero training error and the worst error against the truth

Hold on to

• Capacity beyond the evidence gets spent memorizing noise
• Bias comes from rigidity, variance from sensitivity to the sample
• Verbatim regurgitation by a large model is overfitting at scale

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

1. Reproduce the polynomial demonstration and plot all three fits against the true curve.
   Hint

   Plot all three fits against a dense grid of the true sine, not against the noisy points.

   Solution

   Degree 1 underfits — it cannot bend. Degree 3 tracks the sine closely and is about right. Degree 11 passes near every training point and oscillates violently between them, especially at the edges. Judged against the noisy data it looks best; judged against the truth it is worst. That gap is the whole concept.

   import numpy as np, matplotlib.pyplot as plt
   
   rng = np.random.default_rng(0)
   x = np.linspace(0, 1, 12)
   y = np.sin(2 * np.pi * x) + rng.normal(scale=0.15, size=x.shape)
   xt = np.linspace(0, 1, 400)
   
   plt.scatter(x, y, color="k", zorder=3)
   plt.plot(xt, np.sin(2 * np.pi * xt), "k--", label="truth")
   for degree in (1, 3, 11):
       plt.plot(xt, np.polyval(np.polyfit(x, y, degree), xt), label=f"degree {degree}")
   plt.ylim(-2, 2); plt.legend(); plt.show()

2. Add L2 regularization to the degree-11 fit and find the penalty at which it stops chasing noise.
   Hint

   Ridge shrinks coefficients toward zero; sweep the penalty on a log scale.

   Solution

   Around alpha 1e-3 to 1e-2 the degree-11 fit stops chasing individual points and starts tracking the sine. Note what changed: the hypothesis space still contains the wild oscillating polynomial, but the penalty makes it expensive, so the search no longer selects it. Regularization constrains the search, not the space.

   import numpy as np
   from sklearn.linear_model import Ridge
   from sklearn.preprocessing import PolynomialFeatures
   from sklearn.pipeline import make_pipeline
   
   xt = np.linspace(0, 1, 400)
   truth = np.sin(2 * np.pi * xt)
   
   for alpha in (0, 1e-6, 1e-3, 1e-1, 10):
       model = make_pipeline(PolynomialFeatures(11), Ridge(alpha=max(alpha, 1e-12)))
       model.fit(x.reshape(-1, 1), y)
       error = np.mean((model.predict(xt.reshape(-1, 1)) - truth) ** 2)
       print(f"alpha={alpha:<8} error vs truth={error:.4f}")

Sign in to track your progress through the lab.