Kiru Lab  /  The Mathematics of Learning  /  The Linear Algebra You Actually Need

Decomposition: Finding Structure in a Matrix

Eigenvectors and the SVD tell you what a transformation actually does, independent of coordinates.

Concept  ·  about 30 minutes

An eigenvector of a matrix is a direction the matrix does not rotate — it only stretches it, by a factor called the eigenvalue. Finding those directions tells you the transformation's intrinsic behavior, stripped of whatever arbitrary coordinate system you wrote it in.

W v = lambda v v is an eigenvector, lambda its eigenvalue

The singular value decomposition generalizes this to any matrix, including non-square ones. It factors `W = U S V^T`, where `V^T` rotates the input, `S` stretches each axis by a singular value, and `U` rotates into the output space. Every matrix, without exception, is a rotation, then a stretch, then another rotation.

Reading a layer through its singular values
import numpy as np
U, S, Vt = np.linalg.svd(W, full_matrices=False)
print(S[:10])                       # how much each direction is amplified
print((S > 1e-8).sum(), "effective rank of", min(W.shape))
print(S[0] / S[-1], "condition number")

The condition number

The ratio of largest to smallest singular value tells you how badly the matrix amplifies error. A large condition number means a tiny change in input can produce a large change in output — numerically fragile, slow to optimize, and in a control system, dangerous. This one number connects ill-conditioned optimization landscapes, unstable training, and robot singularities into a single phenomenon.

Structure dictates function

The spectrum of a weight matrix is a structural fact about a layer that you can read without running a single input through it. Low effective rank means the layer is compressing; a heavy tail means it is passing a few dominant directions and ignoring the rest.

Hold on to

  • Every matrix is a rotation, a stretch, and another rotation
  • Singular values reveal effective rank and numerical conditioning
  • Bad conditioning links unstable training and robot singularities

Work through

Try each one before opening the solution. Getting it wrong first is most of where the learning happens.

  1. Compute the SVD of a trained linear layer and plot the singular value spectrum on a log scale. Estimate its effective rank.
    Hint

    Plot `S` from `np.linalg.svd` on a log y-axis.

    Solution

    Trained layers usually show a heavy-tailed spectrum: a few large singular values and a long decaying tail. Effective rank — the count above a tolerance, or the number needed to capture 99% of the energy — is typically far below the nominal dimension, which is why low-rank adaptation methods work at all.

    import numpy as np, matplotlib.pyplot as plt

U, S, Vt = np.linalg.svd(W, full_matrices=False)
plt.semilogy(S); plt.xlabel("index"); plt.ylabel("singular value"); plt.show()

energy = np.cumsum(S ** 2) / np.sum(S ** 2)
print("rank for 99% of energy:", int(np.searchsorted(energy, 0.99)) + 1)
print("nominal rank:", min(W.shape))
  2. Build a matrix with condition number 10^6 and show how a 1e-9 perturbation of the input changes the output.
    Hint

    Build it from a diagonal matrix with singular values 1 and 1e-6.

    Solution

    Solving with the ill-conditioned matrix amplifies the perturbation by roughly the condition number — a 1e-9 input change produces an output change near 1e-3, six orders of magnitude larger. This is the same amplification that makes a robot arm near a singularity demand enormous joint velocities for a tiny hand motion.

    import numpy as np

A = np.diag([1.0, 1e-6])              # condition number 1e6
b = np.array([1.0, 1.0])

x1 = np.linalg.solve(A, b)
x2 = np.linalg.solve(A, b + 1e-9)

print("condition number:", np.linalg.cond(A))       # 1e+06
print("output change:", np.abs(x1 - x2).max())      # ~1e-03
    Check your work

    Paste this after your own code. If it runs without raising, you have it. 

    import numpy as np
assert np.linalg.cond(np.diag([1.0, 1e-6])) > 1e5
print("ok")

Sign in to track your progress through the lab.

← Matrices Are Functions Back to track →