Kiru Lab  /  Classical Machine Learning  /  Classification and Decision Boundaries

Boundaries, Trees, and Choosing a Metric

Different models draw different shapes. Different metrics reward different mistakes.

Hands-On Lab  ·  about 35 minutes

Plot a model's decision boundary on two dimensions and its inductive bias becomes visible. Logistic regression draws a straight line. A decision tree draws axis-aligned rectangles. A k-nearest-neighbor classifier draws a jagged boundary that follows the data. None of these is correct in general; each is correct when its assumption matches the world.

Boundary plotting harness
import numpy as np
import matplotlib.pyplot as plt

def plot_boundary(model, X, y, ax):
    pad = 0.5
    xx, yy = np.meshgrid(
        np.linspace(X[:, 0].min() - pad, X[:, 0].max() + pad, 300),
        np.linspace(X[:, 1].min() - pad, X[:, 1].max() + pad, 300),
    )
    grid = np.c_[xx.ravel(), yy.ravel()]
    zz = model.predict(grid).reshape(xx.shape)
    ax.contourf(xx, yy, zz, alpha=0.25, levels=1)
    ax.scatter(X[:, 0], X[:, 1], c=y, s=12, edgecolor="k", linewidth=0.3)

Metrics encode what you care about

  • Accuracy is useless under class imbalance: 99% of a 99:1 split is achieved by always guessing the majority.
  • Precision asks: of the things I flagged, how many were real?
  • Recall asks: of the real things, how many did I catch?
  • F1 balances the two; a precision-recall curve shows the whole tradeoff rather than one point on it.
  • ROC-AUC is threshold-independent but can look flattering when positives are rare.

Choosing a metric is choosing which mistake you can live with. In a safety context — flagging a model behavior as disordered, say — a false negative and a false positive have very different costs, and the metric should say so out loud rather than hiding it inside an average.

Bridge to Kiru

A DEM-X disorder detector is a classifier. Everything in this lesson applies to it: it has a boundary, a bias, a threshold, and a precision-recall tradeoff that ought to be stated explicitly in any evidence report.

Hold on to

  • A decision boundary is inductive bias made visible
  • Accuracy is the wrong metric under imbalance
  • A detector for AI disorders is itself a classifier with all the same properties

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Plot boundaries for logistic regression, a depth-3 tree, and k-NN on the same two-moons dataset. Explain each shape.
    Hint

    Use `sklearn.datasets.make_moons` and the plotting harness from the lesson.

    Solution

    Logistic regression draws one straight line and cannot follow the crescents, so it misclassifies the interlocking tips. The depth-3 tree draws a staircase of axis-aligned rectangles — better, but visibly boxy where the true boundary is curved. k-NN follows the data closely, including its noise. Each shape is the inductive bias made visible, and none is universally right.

    import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier

X, y = make_moons(n_samples=300, noise=0.2, random_state=0)
models = [LogisticRegression(), DecisionTreeClassifier(max_depth=3),
          KNeighborsClassifier(n_neighbors=5)]

fig, axes = plt.subplots(1, 3, figsize=(13, 4))
for ax, model in zip(axes, models):
    model.fit(X, y)
    plot_boundary(model, X, y, ax)
    ax.set_title(type(model).__name__)
plt.tight_layout(); plt.show()
  2. Build a 98:2 imbalanced dataset and show a model with 98% accuracy and zero recall.
    Hint

    A model that always predicts the majority class scores the majority proportion.

    Solution

    A classifier that predicts the majority class for every input scores 98% accuracy and 0% recall on the minority class — it never once identifies the thing you built it to find. This is why accuracy is the wrong headline metric for rare-event detection, and why any disorder-detection claim in Kiru should report precision and recall on the positive class rather than an overall rate.

    import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, recall_score, precision_score

rng = np.random.default_rng(0)
X = rng.normal(size=(1000, 4))
y = np.zeros(1000, dtype=int); y[:20] = 1            # 98:2 imbalance

model = DummyClassifier(strategy="most_frequent").fit(X, y)
pred = model.predict(X)
print("accuracy:", accuracy_score(y, pred))          # 0.98
print("recall:  ", recall_score(y, pred))            # 0.0
    Check your work

    Paste this after your own code. If it runs without raising, you have it. 

    import numpy as np
from sklearn.metrics import accuracy_score, recall_score
assert abs(accuracy_score(y, pred) - 0.98) < 1e-9
assert recall_score(y, pred) == 0.0
print("ok")

Sign in to track your progress through the lab.

← Logistic Regression and the Softmax Back to track →