Kiru Lab / Classical Machine Learning / Learning From Data
What a Model Actually Is
Three components: a family of functions, a loss, and a search procedure. Nothing else.
Concept · about 25 minutes
Strip away the vocabulary and every supervised learning system has exactly three parts. A hypothesis space: the family of functions you are willing to consider. A loss: a number saying how wrong a particular function is on your data. A search procedure: a way of moving through the hypothesis space toward lower loss.
- Linear regression: lines and planes, squared error, closed-form solution or gradient descent.
- Decision trees: axis-aligned partitions, impurity, greedy splitting.
- A transformer: an enormous parameterized family, cross-entropy, Adam over many steps.
The hypothesis space is where the assumptions live, and they are never neutral. A linear model cannot represent an interaction between two features unless you build that interaction in by hand. A convolutional network assumes that what matters is local and translation-invariant. A transformer assumes that any position may need to consult any other. These commitments are called inductive bias, and choosing one is the most consequential modeling decision you will make.
The architecture is the hypothesis space. Choosing a structure determines which functions are reachable at all — training only selects among the ones the structure already permits. This is the same claim biology makes about anatomy, and it is why the Bio Mirror section belongs next to this one.
Hold on to
- Every model is a hypothesis space, a loss, and a search procedure
- Inductive bias is a commitment made before any data is seen
- Architecture determines what is representable; training only chooses within it
Work through
Try each one before opening the solution. Getting it wrong first is most of where the learning happens.
-
For three models you have used, name the hypothesis space, the loss, and the search procedure explicitly.
Hint
For each model ask: what functions can it express, what number is being minimized, and how is the search performed?
Solution
Worked examples. Random forest: hypothesis space is ensembles of axis-aligned partitions; loss is impurity per split (Gini or entropy); search is greedy recursive splitting with bootstrap resampling. k-NN: hypothesis space is all piecewise-constant functions defined by the training points; there is no loss and no search at all, which is why it is called a lazy learner — the "training" is storage. A fine-tuned transformer: hypothesis space is the pretrained weights plus a small neighborhood; loss is cross-entropy; search is AdamW over a few epochs. Naming the third component is what exposes k-NN as unusual.
-
Construct a dataset a linear model cannot fit at any accuracy, and explain the failure in terms of hypothesis space rather than training.
Hint
XOR, or any target that depends on the product of two features.
Solution
XOR is the canonical case: no straight line separates the classes, so a linear model is stuck at 50% no matter how long you train or how much data you supply. The failure is in the hypothesis space, not the optimizer — adding an interaction feature x1*x2 makes the same linear model solve it instantly, which proves the point. This is the cleanest demonstration that architecture bounds what training can reach.
import numpy as np from sklearn.linear_model import LogisticRegression X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float) y = np.array([0, 1, 1, 0]) # XOR print(LogisticRegression().fit(X, y).score(X, y)) # 0.5 — chance X_aug = np.hstack([X, (X[:, :1] * X[:, 1:])]) # add the interaction print(LogisticRegression().fit(X_aug, y).score(X_aug, y)) # 1.0
Sign in to track your progress through the lab.