Kiru Lab  /  Language Models  /  Text Becomes Numbers

Tokenization: The Model's Sense Organs

A model never sees text. It sees token ids, and the tokenizer decides what distinctions exist.

Concept  ·  about 30 minutes

Before anything else happens, text is split into tokens and each token is mapped to an integer. Modern tokenizers use subword schemes such as byte-pair encoding: frequent words become single tokens, rare words fragment into pieces, and every possible string remains representable because the base alphabet is bytes.

Byte-pair encoding, in miniature
from collections import Counter

def learn_merges(words: list[str], num_merges: int):
    vocab = {" ".join(w) + " </w>": c for w, c in Counter(words).items()}
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for word, freq in vocab.items():
            symbols = word.split()
            for i in range(len(symbols) - 1):
                pairs[(symbols[i], symbols[i + 1])] += freq
        if not pairs:
            break
        best = pairs.most_common(1)[0][0]
        merges.append(best)
        joined = "".join(best)
        vocab = {w.replace(" ".join(best), joined): c for w, c in vocab.items()}
    return merges

Consequences you will meet in practice

  • Character-level tasks are hard: a model asked to count letters in a word is working with a token that may have no internal structure it can inspect.
  • Arithmetic is fragile: numbers tokenize inconsistently, so "1234" and "1235" may share no structure the model can exploit.
  • Non-English text often costs more tokens per unit of meaning, which is a direct cost and context-length penalty for those languages.
  • Rare or adversarial strings fragment into unusual tokens that fall far outside the training distribution — one documented route to strange behavior.
Structure dictates function

The tokenizer is a sense organ, and like any sense organ it determines the boundary of the perceivable. A distinction the tokenizer erases is one no amount of scale can recover. When you catalog a failure in DEM-X, ask whether the tokenizer already explains it before reaching for a higher-level story.

Hold on to

  • Tokenization sets the resolution limit of everything downstream
  • Counting and arithmetic failures often start at the tokenizer
  • Anomalous tokens are a documented source of anomalous behavior

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Tokenize the same paragraph in English and one other language; compare token counts and explain the cost asymmetry.
    Hint

    Use the same tokenizer on a paragraph and its translation, and compare token counts.

    Solution

    English typically costs the fewest tokens per unit of meaning, because tokenizers are trained on corpora dominated by English text and give common English words their own single tokens. Many other languages — especially those not written in Latin script — cost two to four times more tokens for the same content. That is a direct financial cost per API call, a smaller effective context window, and worse performance, all decided before the model sees anything.

    from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")

samples = {
    "english": "The model learns a compressed map of its training data.",
    "other":   "<the same sentence in another language>",
}
for name, text in samples.items():
    ids = tok.encode(text)
    print(f"{name:8} chars={len(text):4} tokens={len(ids):4} "
          f"ratio={len(text)/len(ids):.2f}")
  2. Find a word your tokenizer splits unexpectedly and predict a task the model will fail because of it. Then test that prediction.
    Hint

    Look for a word split into pieces that do not align with its meaning or spelling structure.

    Solution

    A good example is a number: "1234" and "1235" may tokenize into completely different pieces, so the model has no shared structure to generalize arithmetic across. Predict that it will fail at multi-digit addition more often as digit count grows, then test it — this is a real, reproducible prediction derived purely from the tokenizer, and it holds up. Letter-counting tasks fail the same way: the model sees one opaque token, not a sequence of letters.

    from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
for n in ("1234", "1235", "12345", "strawberry"):
    print(n, tok.convert_ids_to_tokens(tok.encode(n)))

Sign in to track your progress through the lab.

← Text Becomes Numbers Embeddings and Position →