Kiru Lab  /  The Mathematics of Learning  /  The Linear Algebra You Actually Need

Matrices Are Functions

A matrix is not a grid of numbers. It is a machine that turns vectors of one kind into vectors of another.

Concept  ·  about 30 minutes

Stop reading a matrix as a table. Read it as a function: `y = Wx` takes a vector `x` in one space and produces a vector `y` in another. The shape tells you the signature. A (10, 64) matrix is a function from a 64-dimensional space to a 10-dimensional one — it can only lose information, never add it.

W: R^64 -> R^10 shape (10, 64) y = W x y has 10 components, each a dot product of x with one row of W

That last observation is worth pausing on. Each row of a weight matrix is a direction, and each output component asks "how much does the input align with my direction?" A neural network layer is a bank of similarity detectors followed by a nonlinearity. That is the entire mechanism.

Composition

Matrix multiplication is function composition. `W2 (W1 x)` applies one map then another, and because it is associative you can precompute `W2 W1` into a single matrix. This is exactly why stacking linear layers without a nonlinearity between them is pointless: the whole stack collapses into one matrix, so the network has no more expressive power than a single layer.

Proving that a linear stack collapses
import numpy as np
rng = np.random.default_rng(0)
W1, W2 = rng.normal(size=(32, 64)), rng.normal(size=(10, 32))
x = rng.normal(size=(64,))

stacked = W2 @ (W1 @ x)
collapsed = (W2 @ W1) @ x
print(np.allclose(stacked, collapsed))   # True — two layers, one function

Rank, and information that cannot come back

The rank of a matrix is the dimension of its output space — how many genuinely independent directions survive the map. A low-rank matrix squashes many inputs onto the same output, and no downstream layer can undo that. Rank is why bottleneck layers force compression, why LoRA finetuning works with so few parameters, and why a robot arm at a singularity cannot move in some direction no matter what the controller asks for.

Forward reference

When the robotics track says "the Jacobian loses rank at a singularity", it means precisely this: an entire direction of desired motion has been mapped to zero, and no amount of joint velocity will produce it.

Hold on to

  • A matrix is a function between vector spaces; its shape is the signature
  • Each row of a weight matrix is a direction the layer detects
  • Rank is how much information survives; lost rank cannot be restored downstream

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Construct a rank-2 matrix in a 5-dimensional space and show numerically that its output always lies in a plane.
    Hint

    Build it as the product of a (5,2) and a (2,5) matrix.

    Solution

    Any product of a tall and a wide matrix through a 2-dimensional waist has rank at most 2, so every output lies in a 2-dimensional subspace no matter what you feed in. Confirm it numerically: the SVD shows only two non-negligible singular values, and outputs from random inputs are all reconstructed exactly from the first two left singular vectors.

    import numpy as np

rng = np.random.default_rng(0)
W = rng.normal(size=(5, 2)) @ rng.normal(size=(2, 5))   # rank 2 by construction

print(np.linalg.matrix_rank(W))          # 2
U, S, Vt = np.linalg.svd(W)
print(S.round(6))                        # only two are non-zero

outputs = np.stack([W @ rng.normal(size=5) for _ in range(50)])
projected = outputs @ U[:, :2] @ U[:, :2].T
assert np.allclose(outputs, projected)   # every output lies in that plane
    Check your work

    Paste this after your own code. If it runs without raising, you have it. 

    import numpy as np
assert np.linalg.matrix_rank(W) == 2
print("ok")
  2. Verify the collapse demonstration above, then insert a ReLU between the layers and show the equality now fails.
    Hint

    Apply ReLU to the intermediate result in the stacked version only.

    Solution

    With the ReLU inserted, `np.allclose` returns False. The collapse proof depended on associativity of matrix multiplication, and a nonlinearity between the two matrices breaks the chain — there is no single matrix equal to the composition. That failure is precisely what makes depth worth having.

    import numpy as np

rng = np.random.default_rng(0)
W1, W2 = rng.normal(size=(32, 64)), rng.normal(size=(10, 32))
x = rng.normal(size=(64,))

assert np.allclose(W2 @ (W1 @ x), (W2 @ W1) @ x)          # linear: collapses

relu = lambda z: np.maximum(0, z)
assert not np.allclose(W2 @ relu(W1 @ x), (W2 @ W1) @ x)  # nonlinear: does not

Sign in to track your progress through the lab.

← Vectors, Dot Products, and Alignment Decomposition: Finding Structure in a Matrix →