Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Estimators, Bias and Variance: Why Unbiased Is Not the Same as Good

7 minute read

Published:

TL;DR: An estimator \(\hat{\theta}\) is a function of the sample, so it is itself random and has a distribution. Mean squared error decomposes exactly as \(\mathrm{MSE} = \mathrm{Var}(\hat\theta) + \mathrm{Bias}(\hat\theta)^2\), with no remainder. Unbiasedness only kills the second term, and it often does so by inflating the first — which is why shrinking an estimate towards zero, as ridge regression does, can lower total error even though it guarantees you are wrong on average.

An estimator is a random variable

An estimator is any function of the sample, \(\hat{\theta} = g(x_1,\dots,x_n)\), used to guess an unknown parameter \(\theta\). The sample mean is one; so is the sample median, the sample maximum, and the constant function that always returns 7. Nothing about the definition requires the estimator to be sensible.

The important consequence is that because the sample is random, \(\hat{\theta}\) is random. It has a distribution — the sampling distribution — and every property below is a property of that distribution, not of the single number you computed from your one dataset.

Four properties

Bias is the systematic offset, \(\mathrm{Bias}(\hat\theta) = \mathbb{E}[\hat\theta] - \theta\). The sample mean is unbiased for \(\mu\). The maximum-likelihood variance \(\frac{1}{n}\sum_i(x_i-\bar{x})^2\) is biased downwards by a factor \((n-1)/n\), as descriptive statistics derives.

Variance is the scatter, \(\mathrm{Var}(\hat\theta) = \mathbb{E}\big[(\hat\theta - \mathbb{E}[\hat\theta])^2\big]\) — how much the answer would move if you collected a different sample of the same size.

Consistency is an asymptotic guarantee: \(\hat\theta_n \to \theta\) in probability as \(n \to \infty\). Consistency and unbiasedness are independent. The MLE of variance is biased at every finite \(n\) yet consistent, since the factor \((n-1)/n \to 1\). Conversely, “use \(x_1\) and ignore the rest” is unbiased for \(\mu\) and never consistent.

Efficiency compares variances among unbiased estimators. The Cramér–Rao bound says no unbiased estimator can beat the inverse Fisher information:

\[ \mathrm{Var}(\hat\theta) \;\ge\; \frac{1}{n \, I(\theta)}, \qquad I(\theta) = -\,\mathbb{E}\!\left[\frac{\partial^2}{\partial\theta^2}\log p(x \mid \theta)\right] \]

Here \(I(\theta)\) is the information a single observation carries about \(\theta\): the expected curvature of the log-density. Sharp curvature means the likelihood changes fast as \(\theta\) moves, so \(\theta\) is easy to pin down. An estimator attaining the bound is called efficient — the sample mean is efficient for a Gaussian mean.

The decomposition

Write \(\bar\theta = \mathbb{E}[\hat\theta]\) and add and subtract it inside the square:

\[ \begin{aligned} \mathrm{MSE}(\hat\theta) &= \mathbb{E}\big[(\hat\theta - \theta)^2\big] = \mathbb{E}\big[\big((\hat\theta - \bar\theta) + (\bar\theta - \theta)\big)^2\big] \\[2pt] &= \underbrace{\mathbb{E}\big[(\hat\theta-\bar\theta)^2\big]}_{\text{variance}} + 2(\bar\theta - \theta)\,\underbrace{\mathbb{E}\big[\hat\theta - \bar\theta\big]}_{=\,0} + \underbrace{(\bar\theta - \theta)^2}_{\text{bias}^2} \end{aligned} \]

The cross term dies because \(\bar\theta-\theta\) is a constant and \(\hat\theta - \bar\theta\) has mean zero by definition. So

\[ \mathrm{MSE}(\hat\theta) = \mathrm{Var}(\hat\theta) + \mathrm{Bias}(\hat\theta)^2 \]

This is an identity, not an approximation and not a trade-off law. It says squared error has exactly two sources, and any estimator that reduces one by more than it inflates the other is an improvement.

The same algebra applied to prediction gives the familiar three-term version. For a target \(y = f(x) + \varepsilon\) with \(\mathrm{Var}(\varepsilon)=\sigma^2\) and a model \(\hat{f}\) trained on a random dataset, the expected squared error at a fixed \(x\) is \(\sigma^2 + \mathrm{Bias}(\hat f(x))^2 + \mathrm{Var}(\hat f(x))\). The extra \(\sigma^2\) is irreducible noise — no model removes it. Underfitting is the bias term dominating; overfitting is the variance term dominating, the model chasing sampling noise that a different training set would not have contained.

Buying a lower error with bias

Suppose \(\bar{x} \sim \mathcal{N}(\mu, \sigma^2/n)\) and consider the shrunk estimator \(\hat\mu_c = c\,\bar{x}\) for a constant \(c \in [0,1]\). Its bias is \((c-1)\mu\) and its variance is \(c^2\sigma^2/n\), so

\[ \mathrm{MSE}(c) = c^2\frac{\sigma^2}{n} + (c-1)^2\mu^2, \qquad c^\star = \frac{\mu^2}{\mu^2 + \sigma^2/n} \]

Take \(\mu = 1\), \(\sigma^2 = 1\), \(n = 4\), so \(\sigma^2/n = 0.25\) and \(c^\star = 1/1.25 = 0.8\).

\(c\)VarianceBias²MSE
1.0 (unbiased)0.25000.00000.2500
0.90.20250.01000.2125
0.8 (optimal)0.16000.04000.2000
0.70.12250.09000.2125
0.50.06250.25000.3125

Shrinking by 20% cuts MSE from 0.2500 to 0.2000 — a 20% reduction — while guaranteeing the estimate is too small on average. The unbiased estimator is not even a local optimum: the derivative of MSE at \(c=1\) is \(2\sigma^2/n > 0\), so some shrinkage always helps.

Variance, squared bias and total MSE for the shrinkage estimatorThree curves plotted against the shrinkage factor c from 0.4 to 1.2, for the case mu equals 1, sigma squared equals 1 and n equals 4. Variance rises as c squared times 0.25, from 0.04 at c equals 0.4 to 0.36 at c equals 1.2. Squared bias falls to zero at c equals 1 and rises again. Their sum, the MSE curve, is a parabola with its minimum of 0.20 at c equals 0.8, which is lower than the value 0.25 attained by the unbiased estimator at c equals 1. 0.40.60.81.01.2 00.20.4 shrinkage factor c (c = 1 is the ordinary unbiased sample mean) MSE 0.200 0.250 variance bias² MSE
Notice that the filled dot (best MSE) is not at c = 1. The MSE curve is flat-bottomed and its minimum sits to the left of the unbiased point, so a little shrinkage is almost free and never harmful near the optimum.
Key Insight — why shrinkage keeps working: the gain comes from the asymmetry of the two terms near \(c=1\). Variance falls linearly in the amount of shrinkage (derivative \(2c\sigma^2/n\), non-zero at \(c=1\)) while squared bias grows only quadratically from zero (derivative \(2(c-1)\mu^2 = 0\) at \(c=1\)). First-order gain against second-order loss means the trade always starts in your favour. Stein's result that the sample mean is inadmissible in three or more dimensions is this observation, made uniform over \(\mu\).

Ridge, and the ML reading

Ridge regression is exactly this move for linear models: \(\hat\beta_{\text{ridge}} = (X^\top X + \lambda I)^{-1}X^\top y\) shrinks every coefficient towards zero, which biases the fit and lowers its variance. When predictors are correlated, \(X^\top X\) is near-singular, the ordinary least-squares variance explodes, and even a large \(\lambda\) pays for itself. Weight decay, early stopping and dropout all sit in the same family. Maximum likelihood shows that the ridge penalty is what a Gaussian prior looks like from the optimisation side.

Interview trap: "bias–variance trade-off" invites two errors. First, the decomposition is an identity, not a constraint — more training data reduces variance without touching bias, so both can fall at once and nothing is being traded. Second, it is specific to squared loss; 0–1 loss and cross-entropy do not decompose this cleanly. And the classic U-shaped test-error curve is not universal: Belkin et al. (2019) documented double descent, where error falls again past the interpolation threshold.

Recap

  • An estimator is a random variable; bias, variance, consistency and efficiency are properties of its sampling distribution.
  • \(\mathrm{MSE} = \mathrm{Var} + \mathrm{Bias}^2\) exactly, and for prediction there is an extra irreducible \(\sigma^2\).
  • Unbiased and consistent are independent properties — the MLE variance is biased but consistent.
  • Shrinking by 20% in the worked example cut MSE by 20%; some shrinkage always helps because variance falls first-order while bias² grows second-order.
  • Ridge, weight decay and early stopping are the same trade in a model with many parameters.

References

  1. Hoerl, A. E., & Kennard, R. W. Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics, 12(1), 55–67, 1970.
  2. Efron, B., & Morris, C. Data Analysis Using Stein’s Estimator and its Generalizations. Journal of the American Statistical Association, 70(350), 311–319, 1975.
  3. Belkin, M., Hsu, D., Ma, S., & Mandal, S. Reconciling modern machine-learning practice and the classical bias–variance trade-off. PNAS, 116(32), 15849–15854, 2019.
  4. Hastie, T., Tibshirani, R., & Friedman, J. The Elements of Statistical Learning, 2nd ed. Springer, 2009.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.