Kiru Lab  /  Foundations: Python as an Instrument  /  Python for Experiments

Reproducibility, Seeds, and Honest Results

An experiment that cannot be re-run is an anecdote. Kiru runs on reproducible evidence.

Concept  ·  about 20 minutes

Randomness enters an experiment in more places than people expect: weight initialization, data shuffling, dropout, augmentation, and — for language models — the sampler at generation time. If you do not control the seed for each of these, you cannot tell the difference between a real improvement and a lucky run.

Pin everything you can
import os, random
import numpy as np

def set_seed(seed: int = 0) -> None:
    random.seed(seed)
    np.random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    try:
        import torch
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.use_deterministic_algorithms(True, warn_only=True)
    except ImportError:
        pass

Seed control is not the same as determinism

GPU kernels reduce floating-point values in nondeterministic order, so bitwise-identical results across machines are often unattainable. The realistic standard is: same seed, same machine, same result; different seeds, results that vary within a band you have measured and reported.

This is the Kiru standard

A DEM-X disorder submission is only evidence if the behavior reproduces across seeds and runs. A single striking screenshot is a hypothesis. Ghostline exists to run the same prompt many times precisely because one sample tells you nothing about a distribution.

  • Record the seed with every result, in the result itself, not in your memory.
  • Report a spread across at least three seeds, not a single number.
  • Log library versions — a silent upgrade changes defaults and therefore results.
  • Separate the seed that controls the model from the seed that controls the data split.

Hold on to

  • Uncontrolled randomness makes an improvement indistinguishable from luck
  • Report a spread across seeds, never a single run
  • Reproducibility is the entry requirement for evidence in Kiru

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Train the same tiny model with five seeds and report mean and range. Decide whether a 1% "improvement" you might read in a paper would be detectable at that spread.
    Hint

    Collect the final metric from each of five runs, then report mean and range.

    Solution

    On a small model the spread across seeds is often 1–3% — which means a reported 1% improvement is indistinguishable from noise unless the paper reports a spread and yours is narrower than the gap. Once you have measured this yourself you will read benchmark tables very differently.

    results = []
for seed in range(5):
    set_seed(seed)
    results.append(train_and_evaluate())

import statistics
print(f"mean {statistics.mean(results):.4f}")
print(f"range {min(results):.4f} - {max(results):.4f}")
print(f"stdev {statistics.stdev(results):.4f}")
  2. Write a `run_metadata()` helper that captures seed, library versions, and git commit, and attach its output to a result file.
    Hint

    Use `importlib.metadata.version` for packages and `subprocess` for the git commit.

    Solution

    Capture it as a dict and write it next to the result, in the same file or directory. Metadata stored anywhere else gets separated from the result it describes, and then the result is an anecdote again.

    import subprocess, sys
from importlib.metadata import version, PackageNotFoundError

def run_metadata(seed, packages=("numpy", "torch")):
    versions = {}
    for name in packages:
        try:
            versions[name] = version(name)
        except PackageNotFoundError:
            versions[name] = None
    try:
        commit = subprocess.check_output(
            ["git", "rev-parse", "HEAD"], text=True, stderr=subprocess.DEVNULL
        ).strip()
    except Exception:
        commit = None
    return {"seed": seed, "python": sys.version.split()[0],
            "packages": versions, "git_commit": commit}
    Check your work

    Paste this after your own code. If it runs without raising, you have it. 

    meta = run_metadata(seed=0, packages=("numpy",))
assert meta["seed"] == 0
assert "python" in meta and "packages" in meta
print("ok")

Related DEM-X entries

GI-DRFT-01

Sign in to track your progress through the lab.

← Seeing Your Data Back to track →