Sampling: The Knobs That Change Everything

import numpy as np

def sample(logits, temperature=1.0, top_k=None, top_p=None, rng=None):
    rng = rng or np.random.default_rng()
    if temperature <= 0:
        return int(logits.argmax())                    # greedy
    logits = logits / temperature                      # <1 sharpens, >1 flattens

    if top_k:
        cutoff = np.partition(logits, -top_k)[-top_k]
        logits = np.where(logits < cutoff, -np.inf, logits)

    probs = np.exp(logits - logits.max())
    probs /= probs.sum()

    if top_p:                                          # nucleus sampling
        order = np.argsort(-probs)
        keep = np.cumsum(probs[order]) <= top_p
        keep[0] = True                                 # always keep the top token
        mask = np.zeros_like(probs, dtype=bool)
        mask[order[keep]] = True
        probs = np.where(mask, probs, 0.0)
        probs /= probs.sum()

    return int(rng.choice(len(probs), p=probs))

Hold on to

• The distribution is the model's output; decoding is your choice
• Sampling settings can dominate model differences
• Decoding configuration is mandatory metadata for evidence

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

1. Generate 20 completions of one prompt at temperature 0.2, 0.7, and 1.2. Describe how the failure mode changes.
   Hint

   Hold the prompt and seed policy fixed; change only the temperature.

   Solution

   At 0.2 outputs are near-identical across runs and failures are consistent — the model makes the same mistake every time. At 0.7 you get variety with occasional drift. At 1.2 outputs diverge sharply and failures shift in character: less repetition, more invented specifics. The failure mode itself changes with temperature, which is why a disorder report without decoding settings is not reproducible.

   for temperature in (0.2, 0.7, 1.2):
       outputs = [generate(prompt, temperature=temperature, seed=s) for s in range(20)]
       unique = len(set(outputs))
       print(f"T={temperature}: {unique}/20 unique")

2. Find a prompt where greedy decoding is correct and sampling at 1.0 is frequently wrong. Quantify the rate.
   Hint

   Look for questions with one correct answer and several plausible-sounding wrong ones.

   Solution

   Factual recall with a plausible distractor is the reliable case: greedy decoding takes the highest-probability token, which is often correct, while sampling at 1.0 sometimes selects a lower-probability but plausible alternative. Report the error rate across at least 50 samples, not a single failure — and note that this makes "the model got it wrong" a statement about a decoding configuration, not about the model alone.

   correct_greedy = sum(check(generate(p, temperature=0)) for p in prompts)
   sampled = [sum(check(generate(p, temperature=1.0, seed=s)) for p in prompts)
              for s in range(5)]
   
   print(f"greedy: {correct_greedy}/{len(prompts)}")
   print(f"sampled: mean {sum(sampled)/len(sampled):.1f}, range {min(sampled)}-{max(sampled)}")

Sign in to track your progress through the lab.