Distributions and Expectation

A probability distribution assigns weight across possible outcomes. Its expectation is the weighted average — the value you would converge to over many draws — and its variance is how far draws typically land from that center.

E[X] = sum_x x * p(x) Var[X] = E[(X - E[X])^2]

Two kinds of uncertainty are worth separating, and conflating them causes real harm. Aleatoric uncertainty is noise inherent in the world: a coin flip stays unpredictable no matter how much data you gather. Epistemic uncertainty is ignorance that more data would fix. A model that reports high confidence on an input unlike anything in its training set is failing to represent epistemic uncertainty.

The mechanical root of a disorder

A softmax always produces a normalized distribution over the allowed outputs. It has no way to say "none of these". That architectural fact — not a training failure — is one reason models produce fluent, confident, wrong answers on out-of-distribution inputs.

Sampling

You will rarely have a distribution in closed form. Instead you draw samples and estimate. This is why running one prompt once tells you almost nothing about a model: you have a single draw from a distribution whose spread you have not measured.

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

import numpy as np, matplotlib.pyplot as plt rng = np.random.default_rng(0) true_mean = np.exp(0.5) # mean of lognormal(0, 1) for n in (10, 100, 10_000): estimates = [rng.lognormal(size=n).mean() for _ in range(500)] print(f"n={n:6d} spread={np.std(estimates):.4f} bias={np.mean(estimates)-true_mean:+.4f}")