Kiru Lab  /  Classical Machine Learning  /  Classification and Decision Boundaries

Logistic Regression and the Softmax

The classifier that never left. Its output layer is the output layer of every language model.

Code Walkthrough  ·  about 35 minutes

Logistic regression computes a weighted sum and then squashes it through a sigmoid to produce a probability. Its multi-class generalization uses a softmax. That softmax over a vocabulary of tokens is precisely the final layer of a large language model — the last step in generating text is a logistic regression over fifty thousand classes.

sigmoid(z) = 1 / (1 + exp(-z)) softmax(z)_i = exp(z_i) / sum_j exp(z_j)
Complete, from scratch
import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-np.clip(z, -500, 500)))

def fit_logistic(X, y, lr=0.1, steps=3000):
    X1 = np.hstack([X, np.ones((len(X), 1))])
    w = np.zeros(X1.shape[1])
    for _ in range(steps):
        p = sigmoid(X1 @ w)
        grad = X1.T @ (p - y) / len(y)   # identical in form to the linear case
        w -= lr * grad
    return w

Notice the gradient. For both squared error on a linear model and cross-entropy on a logistic one, it is `X^T (prediction - target) / n`. That is not a coincidence; it falls out of pairing each likelihood with its natural link function, and it is why the same optimizer code trains both.

Reading coefficients honestly

A logistic coefficient is the change in log-odds per unit of a feature, holding the others fixed. It is interpretable, which is genuinely valuable — but "interpretable" is not "causal". Correlated features split credit arbitrarily between themselves, so a coefficient near zero does not mean the feature is irrelevant.

The interpretability warning, early

This is the first time in the curriculum that a number looks like an explanation and is not one. Hold onto the skepticism. In the mechanistic interpretability track, attention weights will present exactly the same temptation.

Hold on to

  • A language model's output layer is a softmax classifier
  • The same gradient form trains linear and logistic models
  • Interpretable coefficients are not causal claims

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Implement `fit_logistic` and check it against scikit-learn to three decimals.
    Hint

    scikit-learn regularizes by default — set C very large to compare against an unregularized fit.

    Solution

    They agree to three decimals once you disable scikit-learn's default L2 penalty with a large C and give your own descent enough steps. If they disagree by a consistent shrinkage factor, the penalty is the reason, not your gradient.

    import numpy as np
from sklearn.linear_model import LogisticRegression

mine = fit_logistic(X, y, lr=0.5, steps=20000)

reference = LogisticRegression(C=1e9, max_iter=10000).fit(X, y)
theirs = np.append(reference.coef_[0], reference.intercept_[0])

print(np.abs(mine - theirs).max())    # < 1e-3
  2. Duplicate an informative feature and show how the coefficients split. Explain why neither is now readable alone.
    Hint

    Fit once with the original features, once with a duplicated informative column.

    Solution

    The single coefficient of roughly w splits into two of roughly w/2 — the model is indifferent to how credit is divided between identical inputs, so the optimizer's regularization decides. Neither number now answers "how important is this feature?", and with correlated rather than identical features the split is arbitrary rather than even. This is why a small coefficient is not evidence of irrelevance, and it is the first appearance of a lesson that recurs for attention weights and probe directions later.

    import numpy as np
from sklearn.linear_model import LogisticRegression

base = LogisticRegression(C=1e9, max_iter=10000).fit(X, y)
X_dup = np.hstack([X, X[:, :1]])
dup = LogisticRegression(C=1e9, max_iter=10000).fit(X_dup, y)

print("original coef 0:", base.coef_[0][0])
print("split across twins:", dup.coef_[0][0], dup.coef_[0][-1])

Sign in to track your progress through the lab.

← Classification and Decision Boundaries Boundaries, Trees, and Choosing a Metric →