Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Entropy, Cross-Entropy and KL: Why Your Classification Loss Is a Divergence

7 minute read

Published:

TL;DR: Entropy is expected surprise, \(H(p) = -\sum_x p(x)\log p(x)\). Cross-entropy \(H(p,q)\) is the cost of coding \(p\) with a code built for \(q\), and it decomposes as \(H(p) + D_{\mathrm{KL}}(p\|q)\) — so minimising it over \(q\) minimises the KL. KL is non-negative and zero only when \(p=q\), but it is asymmetric and violates the triangle inequality, so it is not a distance. Which argument you put first decides whether your fit covers modes or chases one.

Entropy

Define the surprise of an outcome as $-\log p(x)$: certain outcomes carry none, rare ones carry a lot, and surprises of independent events add because logs turn products into sums. Entropy is the expected surprise:

\[ H(p) = -\sum_x p(x)\log p(x) = \mathbb{E}_{x\sim p}\!\left[-\log p(x)\right]. \]

With $\log_2$ the units are bits. A fair coin has $H = 1$ bit; a coin with $p=0.9$ has $H = -(0.9\log_2 0.9 + 0.1\log_2 0.1) = 0.469$ bits, because most of the time you already knew the answer. Entropy is maximised by the uniform distribution and is zero for a point mass. Shannon’s source coding theorem makes this operational: $H(p)$ is the minimum expected code length, in bits per symbol, for data drawn from $p$.

Cross-entropy and KL

Suppose you build an optimal code for $q$ but the data actually come from $p$. Your expected code length is the cross-entropy

\[ H(p,q) = -\sum_x p(x)\log q(x) = H(p) + D_{\mathrm{KL}}(p\,\|\,q), \qquad D_{\mathrm{KL}}(p\,\|\,q) = \sum_x p(x)\log\frac{p(x)}{q(x)} . \]

The KL divergence is the excess — the bits you waste for using the wrong code. By Jensen’s inequality it is non-negative, and it is zero exactly when $p=q$.

This decomposition explains the classification loss. Take $p$ to be the one-hot label distribution and $q$ the model’s softmax output. Then $H(p) = 0$ — the label is certain — so cross-entropy equals the KL divergence, and it reduces to \(-\log q(y_{\text{true}})\): the negative log-likelihood of the correct class. Training a classifier by cross-entropy is maximum likelihood with a categorical noise model, and it is simultaneously KL minimisation. The three descriptions are one computation.

More generally, maximum likelihood over a dataset of $n$ points is

\[ \arg\max_\theta \frac1n\sum_{i=1}^n \log p_\theta(x_i) \ \approx\ \arg\min_\theta\ D_{\mathrm{KL}}\!\left(p_{\text{data}} \,\|\, p_\theta\right), \]

since the two differ only by \(H(p_{\text{data}})\), a constant in $\theta$. Fitting by likelihood is fitting by forward KL against the empirical distribution.

KL is not a metric

Two failures. It is not symmetric, and it does not satisfy the triangle inequality. Take $p = (0.5, 0.5)$ and $q = (0.9, 0.1)$ on two outcomes:

\[ D_{\mathrm{KL}}(p\|q) = 0.511\ \text{nats} = 0.737\ \text{bits}, \qquad D_{\mathrm{KL}}(q\|p) = 0.368\ \text{nats} = 0.531\ \text{bits}. \]

Different numbers for the same pair. The asymmetry has a clear source: \(D_{\mathrm{KL}}(p\|q)\) averages $\log(p/q)$ under $p$, so it is enormous — infinite, in fact — wherever $p$ puts mass and $q$ puts none, and completely indifferent to regions where $q$ has mass but $p$ does not. Call \(D_{\mathrm{KL}}(p\|q)\) a “distance” in an interview and expect to be asked which direction you meant.

Interview trap — treating KL as a distance: it is asymmetric, unbounded, and can be \(+\infty\) between two perfectly reasonable distributions with different supports. That last point is why GAN training with a KL-like objective gives vanishing gradients when generator and data manifolds do not overlap, and why the Wasserstein distance — a genuine metric that stays finite and informative across disjoint supports — was proposed as a replacement. If you need a symmetric quantity, the Jensen–Shannon divergence symmetrises KL, and its square root is a metric.

Mutual information

Mutual information is the KL between a joint distribution and the product of its marginals:

\[ I(X;Y) = D_{\mathrm{KL}}\bigl(p(x,y) \,\|\, p(x)p(y)\bigr) = H(X) - H(X\mid Y) = H(Y) - H(Y\mid X). \]

It measures how many bits knowing $Y$ saves you when describing $X$. It is symmetric, non-negative, and zero exactly under independence — so, unlike correlation, it detects any dependence, including the parabola counterexample from expectation and variance. The cost is that estimating it from samples in high dimensions is hard, which is why the ML literature is full of variational lower bounds on $I$ rather than direct estimates.

Forward versus reverse KL

Fit a simple \(q_\theta\) to a complicated $p$. The direction you choose changes the answer.

Forward, \(D_{\mathrm{KL}}(p\|q)\) — the expectation is over $p$, so any region where $p>0$ but $q\approx0$ contributes a huge $\log(p/q)$. The fit is zero-avoiding: it spreads $q$ to cover every mode, including the empty space between them. This is the maximum-likelihood direction.

Reverse, \(D_{\mathrm{KL}}(q\|p)\) — the expectation is over $q$, so regions where $q\approx0$ cost nothing regardless of $p$. The penalty falls on placing $q$ mass where $p$ has none. The fit is zero-forcing and mode-seeking: it collapses onto one mode and ignores the rest. This is the direction used by variational inference, because the ELBO is derived from \(D_{\mathrm{KL}}(q\|p)\) — and it is why variational posteriors are famously over-confident and under-dispersed.

Forward versus reverse KL fitting a single Gaussian to a bimodal targetTwo panels. Both show the same bimodal target density with two separated peaks, drawn in grey. In the left panel, labelled forward KL, a single broad Gaussian is centred between the two peaks and spans both, placing mass in the low-density valley. In the right panel, labelled reverse KL, a narrow Gaussian sits on top of the left peak only and ignores the right peak entirely. forward KL(p‖q): mode-covering reverse KL(q‖p): mode-seeking target p (grey), fitted q (colour)
Notice where each fit puts mass it should not. Forward KL fills the valley between the modes — samples from it will look like neither mode. Reverse KL puts nothing there, but silently drops half the target.
Key Insight — the direction encodes what you are willing to be wrong about: forward KL punishes missing mass, so it prefers a blurry model that covers everything — the classic explanation for blurry likelihood-trained generative samples. Reverse KL punishes invented mass, so it prefers a sharp model that covers part of the truth. Neither is more correct; the choice states whether false negatives or false positives are the worse failure for your application.

Where this goes next

The Gaussian’s maximum-entropy property, and the families whose entropies have closed forms, are in common distributions. Estimation and model selection built on likelihood — AIC, likelihood ratios — appear in statistics basics.

Recap

  • \(H(p)\) is expected surprise and the optimal expected code length; a \(p=0.9\) coin carries \(0.469\) bits.
  • \(H(p,q) = H(p) + D_{\mathrm{KL}}(p\|q)\); with one-hot labels \(H(p)=0\), so cross-entropy is exactly the KL and exactly the negative log-likelihood.
  • Maximum likelihood minimises forward KL against the empirical distribution.
  • KL is non-negative and zero only at equality, but asymmetric and not a metric: for \(p=(0.5,0.5)\), \(q=(0.9,0.1)\) the two directions give \(0.511\) and \(0.368\) nats.
  • Forward KL is mode-covering, reverse KL is mode-seeking; variational inference uses the reverse and inherits its over-confidence.

References

  1. Shannon, C. E. A Mathematical Theory of Communication. Bell System Technical Journal 27(3), 379–423, 1948.
  2. Cover, T. M. & Thomas, J. A. Elements of Information Theory, 2nd ed. Wiley, 2006.
  3. Minka, T. Divergence measures and message passing. Microsoft Research Technical Report MSR-TR-2005-173, 2005.
  4. Blei, D. M., Kucukelbir, A. & McAuliffe, J. D. Variational Inference: A Review for Statisticians. JASA 112(518), 859–877, 2017.
  5. Arjovsky, M., Chintala, S. & Bottou, L. Wasserstein GAN. ICML 2017.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.