Kiru Lab  /  Classical Machine Learning  /  Generalization and Its Failures

Splits, Leakage, and Honest Numbers

Most impressive results that fail in production were leaking. Learn the failure patterns.

Concept  ·  about 30 minutes

Leakage is any path by which information from your evaluation set influences the model. It produces results that are excellent and meaningless. It is the single most common cause of a model that performed beautifully in development and collapsed in deployment.

  • Preprocessing leakage: scaling or imputing using statistics computed over the full dataset before splitting.
  • Temporal leakage: training on future rows and testing on past ones when the task is forecasting.
  • Group leakage: the same patient, user, or session appearing on both sides of the split.
  • Selection leakage: choosing features by their correlation with the target measured on all the data.
  • Test-set overfitting: tuning against the same held-out set a hundred times until it stops being held out.
Split first, always
from sklearn.model_selection import train_test_split, GroupKFold
from sklearn.preprocessing import StandardScaler

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

scaler = StandardScaler().fit(X_train)   # fit on train only
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)      # test is transformed, never fitted on

# When rows are grouped (patients, users, documents), split by group:
# GroupKFold(n_splits=5).split(X, y, groups=patient_ids)
Where this bites hardest in LLM work

Benchmark contamination is leakage at web scale: the evaluation questions were in the pretraining corpus. A model can score highly on a reasoning benchmark it has effectively memorized. When you evaluate a model in Kiru, ask first whether your probe could plausibly be in its training data.

Hold on to

  • Split before any fitting, including preprocessing
  • Respect temporal and group structure when splitting
  • Benchmark contamination is leakage in LLM evaluation

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Construct a leaking pipeline and a clean one on the same data; report both scores and the gap.
    Hint

    Fit the scaler on all the data in the leaking version, and on the training split only in the clean one.

    Solution

    On a small dataset the leaking pipeline typically reports a percentage point or two better — small enough to look like a genuine improvement and large enough to decide a model comparison. Leakage through scaling is the mildest form; leakage through feature selection on the full dataset can inflate scores enormously, because you have effectively let the test labels choose your features.

    import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Leaking: scale first, split second
X_leak = StandardScaler().fit_transform(X)
Xtr, Xte, ytr, yte = train_test_split(X_leak, y, test_size=0.3, random_state=0)
print("leaking:", LogisticRegression().fit(Xtr, ytr).score(Xte, yte))

# Clean: split first, fit the scaler on train only
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
scaler = StandardScaler().fit(Xtr)
print("clean:  ", LogisticRegression().fit(scaler.transform(Xtr), ytr)
                    .score(scaler.transform(Xte), yte))
  2. Write a checklist of five leakage questions to ask of any evaluation you are shown.
    Hint

    Think about what was fitted, when it was fitted, and on which rows.

    Solution

    A usable checklist: (1) Was anything fitted — scaler, imputer, encoder, feature selector — before the split? (2) Could a row in test share a group, patient, user, or document with a row in train? (3) If the task is temporal, does every training row precede every test row? (4) Were features constructed using information unavailable at prediction time? (5) How many times has this test set been used to make a decision? Any "yes" to the first four, or a large number to the fifth, means the reported number is optimistic.

Sign in to track your progress through the lab.

← Overfitting and Capacity Back to track →