Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Maximum Likelihood: The Default Recipe for Turning Data Into Parameters

6 minute read

Published:

TL;DR: The likelihood is the sampling density read as a function of the parameter with the data held fixed. Maximising it is equivalent to minimising the KL divergence from the empirical distribution to your model, which is why the negative log-likelihood is the loss function in most of supervised learning. For a Gaussian the MLE gives \(\hat\mu = \bar{x}\) and \(\hat\sigma^2 = \frac{1}{n}\sum_i (x_i - \bar{x})^2\) — and that variance estimate is biased low. Adding a prior turns MLE into MAP, and a Gaussian prior is exactly an L2 penalty.

Likelihood is the density, read sideways

For a model \(p(x \mid \theta)\) and independent observations \(x_1,\dots,x_n\), the likelihood is

\[ L(\theta) = \prod_{i=1}^{n} p(x_i \mid \theta) \]

Algebraically this is the joint density. Conceptually it is something different: the data are fixed and known, and \(\theta\) is the free argument. \(L(\theta)\) is not a probability distribution over \(\theta\) — it does not integrate to one, and rescaling it changes nothing about where the maximum is. Only ratios of likelihoods at different \(\theta\) carry meaning.

We maximise the logarithm because sums are easier than products in three separate ways: the derivative of a sum is a sum of derivatives; the exponential families that dominate practice have log-densities that are simple polynomials; and a product of \(10^5\) densities underflows to zero in floating point while its log does not. Since \(\log\) is strictly increasing, the maximiser is unchanged.

The Gaussian, worked end to end

Take \(x_i \sim \mathcal{N}(\mu, \sigma^2)\) with both parameters unknown. The log-likelihood is

\[ \ell(\mu, \sigma^2) = -\frac{n}{2}\log(2\pi) - \frac{n}{2}\log \sigma^2 - \frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \mu)^2 \]

Differentiate in \(\mu\): only the last term depends on it, giving \(\partial\ell/\partial\mu = \frac{1}{\sigma^2}\sum_i (x_i - \mu)\). Setting this to zero gives \(\hat\mu = \bar{x}\), independently of \(\sigma^2\).

Now differentiate in \(\sigma^2\), treating it as a single variable \(v\):

\[ \frac{\partial \ell}{\partial v} = -\frac{n}{2v} + \frac{1}{2v^2}\sum_i (x_i - \hat\mu)^2 = 0 \qquad\Longrightarrow\qquad \hat\sigma^2 = \frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2 \]

Concretely, for the five measurements 2.1, 1.9, 2.4, 2.0, 2.6: the sum is 11.0 so \(\hat\mu = 2.2\), and the squared deviations \(0.01 + 0.09 + 0.04 + 0.04 + 0.16\) total 0.34. The MLE gives \(\hat\sigma^2 = 0.34/5 = 0.068\), while the unbiased estimator gives \(0.34/4 = 0.085\). The ratio is \(0.8 = (n-1)/n\), exactly as the Bessel argument in descriptive statistics predicts.

Interview trap — "is the MLE biased?": for the Gaussian mean, no. For the variance, yes: \(\mathbb{E}[\hat\sigma^2] = \frac{n-1}{n}\sigma^2\), an underestimate at every finite \(n\). The reason is that the MLE divides by \(n\) while measuring deviations from the fitted \(\bar{x}\) rather than the true \(\mu\). Maximum likelihood is consistent and asymptotically efficient under regularity conditions, but it carries no finite-sample unbiasedness guarantee — and unbiasedness is not preserved by reparameterisation, whereas the MLE is (the MLE of \(\sigma\) is \(\sqrt{\hat\sigma^2}\)).

Curvature is information

Under regularity conditions the MLE is asymptotically normal:

\[ \sqrt{n}\,(\hat\theta_n - \theta) \;\xrightarrow{d}\; \mathcal{N}\!\big(0,\ I(\theta)^{-1}\big), \qquad I(\theta) = -\,\mathbb{E}\!\left[\frac{\partial^2 \log p(x\mid\theta)}{\partial\theta^2}\right] \]

\(I(\theta)\) is the Fisher information per observation — the expected curvature of the log-density at the truth. A sharply peaked log-likelihood means small parameter changes cost a lot of fit, so the parameter is well determined; a flat one means many values fit almost equally well. For the Gaussian mean, \(I(\mu) = 1/\sigma^2\), so the asymptotic variance of \(\hat\mu\) is \(\sigma^2/n\) — which here is the exact finite-sample answer too.

Log-likelihood curvature at two sample sizesTwo log-likelihood curves for a Gaussian mean, both peaking at mu equals 2.2 and both normalised to zero at the peak. The curve for n equals 5 is broad, falling only about 2.5 log units by mu equals 1.2. The curve for n equals 20 is four times as curved and falls off the bottom of the plot within about 0.7 of the peak. Standard error markers show plus or minus 0.447 for n equals 5 and plus or minus 0.224 for n equals 20, illustrating that four times the data gives four times the curvature and halves the standard error. 1.21.72.22.73.2 0−2−4 candidate mean μ (peak at the MLE, μ̂ = x̄ = 2.2); vertical axis: log-likelihood relative to the peak ±0.447 ±0.224 n = 5 (flat) n = 20 (sharp)
Notice that the peak location is the same in both curves — more data does not move the estimate, it narrows the set of parameters that explain the data almost as well. Curvature at the peak is the information, and its reciprocal is the squared standard error.

The same thing as minimising KL

Let \(\hat{p}\) be the empirical distribution, putting mass \(1/n\) on each observed point. Then

\[ D_{\mathrm{KL}}\!\left(\hat{p}\,\|\,p_\theta\right) = \underbrace{\mathbb{E}_{\hat p}[\log \hat p(x)]}_{\text{no }\theta} - \frac{1}{n}\sum_{i=1}^n \log p(x_i \mid \theta) \]

The first term is the negative entropy of the data and does not involve \(\theta\). So minimising KL divergence to the empirical distribution and maximising the average log-likelihood are the same optimisation. This is the bridge to machine learning: cross-entropy loss on a classifier is the negative log-likelihood of a categorical model, and squared error is the negative log-likelihood of a Gaussian with fixed variance. Nobody chose those losses for aesthetic reasons.

MAP, and why L2 is a Gaussian prior

Put a prior \(p(\theta)\) on the parameter and maximise the posterior instead. Since \(p(\theta \mid x) \propto p(x\mid\theta)\,p(\theta)\),

\[ \hat\theta_{\mathrm{MAP}} = \arg\max_\theta \Big[ \log p(x \mid \theta) + \log p(\theta) \Big] \]

MAP is MLE plus one extra additive term. Take linear regression with Gaussian noise of variance \(\sigma^2\) and a prior \(\theta \sim \mathcal{N}(0, \tau^2 I)\). The log-prior is \(-\lVert\theta\rVert^2/(2\tau^2)\) up to a constant, so maximising gives

\[ \arg\min_\theta \ \sum_i (y_i - x_i^\top\theta)^2 \;+\; \frac{\sigma^2}{\tau^2}\lVert\theta\rVert^2 \]

which is ridge regression with \(\lambda = \sigma^2/\tau^2\). A tight prior (small \(\tau\)) means heavy regularisation; a flat prior recovers plain MLE. A Laplace prior gives L1 and hence lasso by the same route.

Key Insight — MAP is not Bayesian inference: MAP returns the mode of the posterior, a single point, and discards the rest of the distribution. That makes it reparameterisation-dependent — the mode of a density moves under a nonlinear change of variables while the posterior itself does not — and it says nothing about uncertainty. Genuine Bayesian inference keeps the whole posterior; see Bayesian versus frequentist. MAP is best understood as regularised optimisation wearing Bayesian notation.

Recap

  • Likelihood is the density with data fixed and \(\theta\) free; it is not a distribution over \(\theta\), so only ratios matter.
  • Gaussian MLE: \(\hat\mu = \bar{x}\) (unbiased), \(\hat\sigma^2 = \frac1n\sum_i(x_i-\bar x)^2\) (biased by \((n-1)/n\)).
  • Asymptotically \(\hat\theta \approx \mathcal{N}(\theta, (nI(\theta))^{-1})\); Fisher information is expected log-likelihood curvature.
  • Maximising log-likelihood = minimising \(D_{\mathrm{KL}}(\hat p \,\|\, p_\theta)\), which is why cross-entropy and squared error are the standard losses.
  • MAP = MLE + log-prior; a Gaussian prior gives ridge with \(\lambda = \sigma^2/\tau^2\), a Laplace prior gives lasso.

References

  1. Fisher, R. A. On the Mathematical Foundations of Theoretical Statistics. Philosophical Transactions of the Royal Society A, 222, 309–368, 1922.
  2. van der Vaart, A. W. Asymptotic Statistics. Cambridge University Press, 1998.
  3. Efron, B. R. A. Fisher in the 21st Century. Statistical Science, 13(2), 95–122, 1998.
  4. Murphy, K. P. Probabilistic Machine Learning: An Introduction. MIT Press, 2022.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.