Kiru Lab  /  Language Models

Text Becomes Numbers

Tokenization and embeddings — the two steps that decide what a model can even perceive.

In this module you will

  • Explain subword tokenization and its consequences
  • Describe an embedding space and how position is injected
Tokenization: The Model's Sense Organs
A model never sees text. It sees token ids, and the tokenizer decides what distinctions exist.
Concept · 30m
Embeddings and Position
A token id becomes a learned vector, and position gets injected because attention alone is order-blind.
Concept · 30m
← Language Models