PCA: Maximum Variance and Minimum Reconstruction Error
Published:
The same answer to two questions
Question one: maximum variance. Find the unit vector \(u\) along which the projected data \(Xu\) has the largest variance. That variance is \(u^\top \Sigma u\) where \(\Sigma\) is the covariance matrix, so you want
whose solution is the top eigenvector of \(\Sigma\), with the maximum value equal to the top eigenvalue.
Question two: minimum reconstruction error. Find the \(k\)-dimensional subspace such that projecting the data onto it loses as little as possible in squared distance.
These sound like different problems. They are the same problem, and the reason is one line of Pythagoras. For a centred point \(x_i\) and an orthogonal projection \(P\),
The left side does not depend on \(P\). So maximising the first term on the right and minimising the second are the same optimisation, and they must select the same subspace.
The mechanics, on numbers you can check
Take five points, already centred:
The covariance is \(\Sigma = \tfrac{1}{n-1}X^\top X\) with \(n = 5\):
For a \(2 \times 2\) matrix the eigenvalues follow from the trace and determinant: \(\operatorname{tr}\Sigma = 3.5\) and \(\det\Sigma = 2.5 - 2.25 = 0.25\), so
They sum to \(3.5\), matching the trace, and multiply to \(0.25\), matching the determinant. The first component carries \(3.4271/3.5 = 97.9\%\) of the variance, and its eigenvector is \(u_1 \approx (0.851,\ 0.526)\).
Now check the equivalence claim numerically. Projecting onto \(u_1\) alone leaves a total squared reconstruction error of \(0.2918\). And \((n-1)\lambda_2 = 4 \times 0.0729 = 0.2918\). The variance you discard is the error you incur โ exactly, not approximately.
Use the SVD, not the covariance matrix
Everything above is easier and better conditioned through the singular value decomposition \(X = U S V^\top\). The right singular vectors are the principal directions, and the eigenvalues follow from the singular values:
On the example, \(s = (3.7025,\ 0.5402)\), and \(s^2/4 = (3.4271,\ 0.0729)\) โ the eigenvalues again.
Two preprocessing steps that are not optional
Centring. PCA on uncentred data does not find the direction of greatest spread โ it finds the direction of the mean. Shift the example by \((10,10)\) and the leading direction becomes \(\approx(0.709, 0.705)\), which points at the centroid, not along the data. The variance structure is unchanged; only the centring was dropped. Most libraries centre for you. Verify that yours does.
Scaling. Covariance is not unit-free. Measure one feature in metres and another in millimetres and the millimetre featureโs variance is a million times larger, so PC1 will align with it regardless of whether it carries any real information. If your features have incomparable units, standardise first โ which amounts to running PCA on the correlation matrix instead.
How many components
The honest answer is that this is a judgement call dressed up in several ways: keep enough components to reach a chosen cumulative explained-variance ratio (90%, 95%); look for a bend in the scree plot; or, if you are feeding a downstream model, cross-validate the number as a hyperparameter. The last is the only one that optimises something you actually care about. The first two are conventions, and the โelbowโ is frequently not there.
What PCA cannot do
It is linear. The components are linear combinations of the original features, so PCA can rotate and project but never bend. Data on a spiral, a Swiss roll, or a circle has no low-dimensional linear description, and PCA will report that honestly by spreading variance across many components. That is not a failure of the algorithm; it is the algorithm telling you its model does not fit.
Variance is not importance. PCA is unsupervised and has never seen your labels. A direction with tiny variance can be the one that separates your classes, and PCA will discard it first. If you have labels and want a discriminative projection, that is a different method.
Components are rarely interpretable. PC1 is a weighted mixture of every original feature. Occasionally that mixture means something; usually it does not, and reading a story into the loadings is a common way to fool yourself.
Non-linear alternatives, and a warning. Kernel PCA applies the kernel trick to do PCA in a feature space. Autoencoders learn a non-linear encoder and decoder, and reduce to PCAโs subspace when made linear with a squared-error loss. t-SNE and UMAP are for visualisation.
โ Key Takeaways
- Maximising retained variance and minimising squared reconstruction error are the same optimisation โ Pythagoras splits a fixed total between them.
- Components are eigenvectors of the covariance; explained variance is its eigenvalues. On the worked example \(\lambda = (3.4271, 0.0729)\), so PC1 carries 97.9%.
- The discarded variance equals the reconstruction error: \((n-1)\lambda_2 = 0.2918\) is exactly the SSE from projecting onto PC1.
- Compute via SVD, using \(\lambda_i = s_i^2/(n-1)\). Forming \(X^\top X\) squares the condition number for no benefit.
- Centre always; standardise whenever features have different units. Skipping the centring makes PC1 point at the mean.
- PCA is linear, unsupervised, and optimises reconstruction โ so it cannot unroll curved structure, and it has no reason to keep the direction your classifier needs.