Kiru Lab / Track

Language Models

A next-token predictor, examined until the surprising parts stop being surprising.

This track dismantles the transformer piece by piece: how text becomes numbers, how attention routes information between positions, what a residual stream is, and what pretraining and alignment each actually change. It ends where Kiru begins — with the mechanical origins of confabulation, sycophancy, and drift, which are not bugs bolted onto the architecture but consequences of it.

The question this track answers

How does predicting the next token produce behavior that looks like reasoning?

By the end you can

Trace a token from input string to output probability
Implement scaled dot-product attention from scratch
Explain the residual stream as a shared communication channel
Describe what RLHF changes and what it cannot change
Give a mechanical account of confabulation and sycophancy

Come here from

Deep Learning

Scope

7 lessons, roughly 24 hours.

Module 1

Text Becomes Numbers

Tokenization and embeddings — the two steps that decide what a model can even perceive.

Tokenization: The Model's Sense Organs

A model never sees text. It sees token ids, and the tokenizer decides what distinctions exist.

Embeddings and Position

A token id becomes a learned vector, and position gets injected because attention alone is order-blind.

Module 2

Anatomy of a Transformer

Attention, the residual stream, and the block that repeats until you have a model.

Self-Attention, Built From Scratch

Queries, keys, values — a differentiable lookup table. You already know every operation involved.

Hands-On Lab · 50m

The Residual Stream and the Transformer Block

The most useful mental model in interpretability: a shared bus that every layer reads from and writes to.

Module 3

Training, Behavior, and Failure

Pretraining, alignment, sampling — and the mechanical origins of the disorders DEM-X catalogs.

Pretraining, Finetuning, and Alignment

Three stages with three different objectives. Most behavioral surprises live in the gaps between them.

Sampling: The Knobs That Change Everything

The model outputs a distribution. What you see is a sample from it, and the sampler is a variable you must control.

Code Walkthrough · 30m

Why Models Confabulate

The bridge into DEM-X. Confabulation is what this architecture does when it lacks the answer.

← Deep Learning Embodied Control: Kinematics and the Inverse Jacobian →