Bias, Variance, and Regularisation: Why Models Fail to Generalise
Published:
The decomposition, derived
Suppose the data comes from \(y = f(x) + \varepsilon\) with \(\mathbb{E}[\varepsilon] = 0\) and \(\operatorname{Var}(\varepsilon) = \sigma^2\). You fit \(\hat{f}\) on a random training sample, so \(\hat{f}\) is itself random. Fix a test point \(x\) and ask for the expected squared error there, averaging over both the training sample and the fresh noise.
Add and subtract the average prediction \(\bar{f}(x) = \mathbb{E}[\hat{f}(x)]\):
The cross term disappears because \(\bar{f}(x) - f(x)\) is a constant with respect to the randomness in \(\hat{f}\), so it factors out of an expectation that is zero: \(\mathbb{E}[\hat{f}(x) - \bar{f}(x)] = 0\). The noise term separates because \(\varepsilon\) is independent of the training sample.
The two extremes make it concrete. A model that always predicts the constant \(0\) has zero variance — resampling changes nothing — and enormous bias. A 1-nearest-neighbour fit has almost no bias, since it can represent any function given enough points, and large variance, since moving one training point changes its predictions wholesale.
Measured, not asserted
Fit polynomials of increasing degree to ten noisy samples of a smooth function, and evaluate on ten held-out points.
| Degree | Train RMSE | Validation RMSE |
|---|---|---|
| 0 | 0.6456 | 0.7503 |
| 1 | 0.4527 | 0.4899 |
| 2 | 0.1316 | 0.2681 |
| 3 | 0.0853 | 0.2171 |
| 5 | 0.0816 | 0.2123 |
| 7 | 0.0380 | 0.2402 |
| 8 | 0.0238 | 0.2558 |
| 9 | ≈ 0 | 0.2915 |
Decomposing the error directly at a single test point, over 4,000 resampled training sets:
| Degree | bias² | variance | sum | measured error |
|---|---|---|---|---|
| 0 | 0.35011 | 0.00379 | 0.35391 | 0.35391 |
| 3 | 0.00667 | 0.00882 | 0.01549 | 0.01549 |
| 9 | 0.00004 | 0.02924 | 0.02928 | 0.02928 |
The identity holds to every digit shown, and the pattern is exactly as advertised: the constant-ish model is nearly all bias, the interpolating model is nearly all variance, and the middle is where their sum is smallest.
Train, validation, test
Three splits, three jobs. Training fits parameters. Validation chooses hyperparameters — degree, \(\lambda\), depth, learning rate. Test estimates generalisation, once.
Cross-validation gets more out of limited data. Split into \(k\) folds, train on \(k-1\) and validate on the remaining one, rotate, average. Every point is used for validation exactly once, so the estimate is far less sensitive to an unlucky split than a single hold-out, at \(k\) times the compute.
Regularisation is buying variance with bias
Every technique below deliberately introduces bias because the variance reduction more than pays for it.
Ridge adds \(\lambda\lVert \beta \rVert_2^2\) and shrinks coefficients smoothly towards zero. Lasso adds \(\lambda\lVert \beta \rVert_1\) and sets some exactly to zero, giving selection as well as shrinkage. Both are derived in the linear regression chapter, including the geometric reason only lasso produces sparsity.
Early stopping halts training when validation error turns up. It is regularisation by wall clock: you never reach the high-variance region of parameter space because you stop first.
Dropout randomly zeroes units during training, so no unit can rely on any particular other one being present.
Data augmentation is the odd one out, and the most effective when it applies. Instead of restricting the model, it enlarges the data with transformations that should not change the label. That reduces variance without adding bias — provided your invariance claim is true. Rotating handwritten digits by 180° is not label-preserving; a 6 becomes a 9.
Where the classical picture stops
The U-shaped validation curve above is real and reproduces reliably at this scale. It is not the complete story.
For very heavily over-parameterised models, test error often falls again after the point where the model has enough capacity to interpolate the training set — the phenomenon usually called double descent. The classical account predicts monotone deterioration past the interpolation threshold, and that is not what is observed for large neural networks.
The bias–variance decomposition itself remains exactly true — it is an algebraic identity, not an empirical claim. What double descent complicates is the assumption that variance rises monotonically with parameter count.
✅ Key Takeaways
- Expected squared error \(=\ \text{bias}^2 + \text{variance} + \sigma^2\). The cross term vanishes because \(\bar{f} - f\) is constant with respect to the training randomness. This is an identity, not an approximation.
- \(\sigma^2\) is irreducible. A held-out error below the noise level is evidence of leakage, not of a good model.
- Measured on real fits: degree 0 gives bias² 0.350 and variance 0.004; degree 9 gives bias² 0.00004 and variance 0.029. Same total error budget, opposite causes.
- Training error falls monotonically with capacity and therefore carries no information about generalisation. Only held-out error does.
- A test set consulted repeatedly stops being one — the leakage is through your decisions, not your weights.
- Ridge, lasso, early stopping and dropout all buy variance reduction with bias. Data augmentation is the exception, reducing variance for free — if the invariance you assume is genuinely true.
- Double descent means "more parameters than data" does not by itself imply overfitting. Look at the curve.