Kiru Lab  /  The Mathematics of Learning  /  The Linear Algebra You Actually Need

Vectors, Dot Products, and Alignment

The dot product is the single most-used operation in machine learning. It measures agreement.

Concept  ·  about 30 minutes

A vector is an ordered list of numbers, and in this field it is almost always a position in a space of meaning. The word "king" in a language model is a point in a 4096-dimensional space; a joint configuration of a robot arm is a point in a space with one dimension per joint; an image is a point in a space with one dimension per pixel. Same object, different rooms.

The dot product

a · b = sum_i a_i * b_i = |a| |b| cos(theta)

Read the right-hand side. The dot product is large when two vectors point the same way, zero when they are perpendicular, and negative when they oppose. It is a similarity score, and when you normalize away the magnitudes you get cosine similarity — the standard way to ask a model whether two things mean roughly the same thing.

Similarity in eight lines
import numpy as np

def cosine(a: np.ndarray, b: np.ndarray) -> float:
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

king, queen, carburetor = embeddings["king"], embeddings["queen"], embeddings["carburetor"]
print(cosine(king, queen))        # ~0.7 — close in meaning space
print(cosine(king, carburetor))   # ~0.05 — unrelated directions
Where this reappears

Attention scores are dot products between a query vector and every key vector, scaled and softmaxed. When you learn attention in the LLM track, you will already know the operation — the only new part is what the vectors mean.

Norms and distance

The L2 norm is the ordinary length of a vector; the L1 norm is the sum of absolute values. The choice between them shows up as the difference between ridge and lasso regularization, between smooth and sparse solutions, and — much later — between dense and sparse feature dictionaries in interpretability.

Hold on to

  • A vector is a position in a space of meaning
  • The dot product measures alignment and is the core of attention
  • Norm choice determines whether solutions come out smooth or sparse

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Implement cosine similarity and verify that a vector is perfectly similar to itself and opposite to its negation.
    Hint

    cos(0) is 1 and cos(180°) is -1.

    Solution

    A vector is perfectly aligned with itself (1.0) and perfectly opposed to its negation (-1.0). The guard against a zero-magnitude vector matters: the zero vector has no direction, so cosine similarity is undefined for it, and without the guard you get a silent NaN that propagates through everything downstream.

    import numpy as np

def cosine(a, b):
    na, nb = np.linalg.norm(a), np.linalg.norm(b)
    if na == 0 or nb == 0:
        raise ValueError("cosine similarity is undefined for the zero vector")
    return float(a @ b / (na * nb))
    Check your work

    Paste this after your own code. If it runs without raising, you have it. 

    import numpy as np
v = np.array([1.0, 2.0, 3.0])
assert abs(cosine(v, v) - 1.0) < 1e-9
assert abs(cosine(v, -v) + 1.0) < 1e-9
assert abs(cosine(np.array([1.0, 0.0]), np.array([0.0, 1.0]))) < 1e-9
print("ok")
  2. Take 20 word embeddings and print the nearest neighbor of each. Find one result that is wrong and explain what the geometry did.
    Hint

    Look for neighbors that are related by co-occurrence rather than by meaning.

    Solution

    The usual failures: antonyms come out close ("hot" and "cold" appear in identical contexts, so the geometry cannot separate them), frequent words act as attractors for everything, and a polysemous word like "bank" lands between its senses rather than near either. The lesson is that the space encodes distributional similarity — which contexts a word appears in — not meaning, and those diverge exactly where you would most want them not to.

    import numpy as np

def nearest(word, embeddings, k=1):
    target = embeddings[word]
    scored = [(cosine(target, vec), other)
              for other, vec in embeddings.items() if other != word]
    return sorted(scored, reverse=True)[:k]

In the Bio Mirror

Population Coding

Sign in to track your progress through the lab.

← The Linear Algebra You Actually Need Matrices Are Functions →