intuition.
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
Linear Algebra
Step 22 of 22

PCA: the destination

Understand first: symmetric matrices

A cloud of data points. Each is a vector — measurements of something, one number per feature.

Look at the cloud in the visual. It is clearly stretched: it varies a lot along one diagonal direction and barely at all across it. Those axes are not ı^\hat{\imath} and ȷ^\hat{\jmath}. The data does not care what basis you happened to start with.

Measuring the spread

To find those directions, first measure how the variables move together. For each pair of features, compute how they vary with respect to one another, and collect the answers in the covariance matrix:

C=[var(x)cov(x,y)cov(x,y)var(y)]C = \begin{bmatrix} \text{var}(x) & \text{cov}(x,y) \ \text{cov}(x,y) & \text{var}(y) \end{bmatrix}

Notice the shape. The off-diagonal entries are equal, because the covariance of xx with yy is the covariance of yy with xx. The covariance matrix is symmetric, always, by construction.

Which triggers the guarantee from step 21: real eigenvalues, perpendicular eigenvectors, always diagonalizable.

The claim, and why it holds

Take the eigenvectors of CC. They point along the directions the data varies most, and each eigenvalue is exactly the amount of variance along its eigenvector.

Here is the reason, and it is the payoff of everything before it. Diagonalizing CC means finding the basis in which it is diagonal — a basis where all the off-diagonal entries vanish. Off-diagonal entries are covariances, so making them zero means finding directions along which the data has no covariance at all: the coordinates become uncorrelated. Each new axis carries pure, independent variation, and the eigenvalue attached to it is how much.

That is principal component analysis. The eigenvector with the largest eigenvalue is the first principal component: the single direction that captures more of the data than any other.

Why it is useful

Keep the components with large eigenvalues, discard the ones with small eigenvalues, and you have thrown away dimensions while losing almost nothing — because the discarded directions genuinely held almost no variation. A thousand-feature dataset can often be squeezed into a handful of components with most of its structure intact.

Look back

You started with an arrow on a grid, and the decision to call it a pair of numbers.

Every step since was forced by the one before: two operations, what they reach, what makes a good basis, moving all of space, recording the motion in columns, measuring what it stretches and destroys, and finally asking which directions a transformation leaves alone.

PCA is not a formula that appeared from nowhere. It is that same question asked of data, and you now know exactly why the answer is what it is.