The Ten Distributions You Need, and the Geometry of the Multivariate Gaussian
Published:
The reference table
\(\operatorname{Var}\) is the variance of a single draw; parameters are on the left.
| Distribution | Models | Parameters | Mean | Variance |
|---|---|---|---|---|
| Bernoulli\((p)\) | one binary trial | \(p\in[0,1]\) | \(p\) | \(p(1-p)\) |
| Binomial\((n,p)\) | successes in \(n\) trials | \(n\in\mathbb{N},\,p\) | \(np\) | \(np(1-p)\) |
| Categorical\((\boldsymbol{\pi})\) | one draw from \(K\) classes | \(\boldsymbol{\pi}\), \(\sum_k\pi_k=1\) | \(\boldsymbol{\pi}\) | \(\operatorname{diag}(\boldsymbol{\pi})-\boldsymbol{\pi}\boldsymbol{\pi}^{\top}\) |
| Poisson\((\lambda)\) | counts of rare independent events | \(\lambda>0\) | \(\lambda\) | \(\lambda\) |
| Uniform\((a,b)\) | no preference on an interval | \(a \lt b\) | \(\frac{a+b}{2}\) | \(\frac{(b-a)^2}{12}\) |
| Gaussian\((\mu,\sigma^2)\) | sums of many small effects | \(\mu,\ \sigma^2>0\) | \(\mu\) | \(\sigma^2\) |
| Exponential\((\lambda)\) | memoryless waiting time | \(\lambda>0\) | \(1/\lambda\) | \(1/\lambda^2\) |
| Beta\((\alpha,\beta)\) | an unknown probability | \(\alpha,\beta>0\) | \(\frac{\alpha}{\alpha+\beta}\) | \(\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}\) |
| Dirichlet\((\boldsymbol{\alpha})\) | an unknown probability vector | \(\alpha_k>0\), \(\alpha_0=\sum_k\alpha_k\) | \(\alpha_k/\alpha_0\) | \(\frac{\alpha_k(\alpha_0-\alpha_k)}{\alpha_0^2(\alpha_0+1)}\) |
| Laplace\((\mu,b)\) | sharply peaked, heavy-tailed noise | \(\mu,\ b>0\) | \(\mu\) | \(2b^2\) |
Three relationships tie the table together. Binomial is a sum of independent Bernoullis. Poisson is the limit of Binomial\((n,\lambda/n)\) as \(n\to\infty\), which is why it models rare events among many opportunities. Beta is the one-dimensional case of Dirichlet.
The three that matter most in ML
Categorical. Every classifier’s output layer. A softmax over \(K\) logits is a categorical parameter vector, and the training loss is the negative log-likelihood of the observed class under it. Its covariance \(\operatorname{diag}(\boldsymbol{\pi})-\boldsymbol{\pi}\boldsymbol{\pi}^{\top}\) is exactly the Jacobian of the softmax, which is why the softmax–cross-entropy gradient collapses to \(\hat{\mathbf{y}} - \mathbf{y}\).
Beta and Dirichlet. These are the conjugate priors for Bernoulli and Categorical: start with Beta\((\alpha,\beta)\), observe \(s\) successes and \(f\) failures, and the posterior is Beta\((\alpha+s,\beta+f)\). The parameters act as pseudo-counts, which is the cleanest way to read Laplace smoothing, adding 1 to every count is a Dirichlet\((\mathbf{1})\) prior. Dirichlet is also what latent Dirichlet allocation puts over topic proportions.
Laplace. Its log-density is \(-\lvert x-\mu\rvert/b\) up to a constant, so a Laplace prior on weights gives an \(\ell_1\) penalty and a Gaussian prior gives \(\ell_2\). Lasso versus ridge is a choice of noise model, not just a choice of penalty.
The multivariate Gaussian
For \(\mathbf{x}\in\mathbb{R}^d\) with mean \(\boldsymbol{\mu}\) and positive-definite covariance \(\boldsymbol{\Sigma}\):
Everything about the shape is in the quadratic form. The quantity \(\Delta^2 = (\mathbf{x}-\boldsymbol{\mu})^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu})\) is the squared Mahalanobis distance, and the density is constant wherever \(\Delta\) is constant, so the level sets are ellipsoids.
Diagonalise, \(\boldsymbol{\Sigma} = \mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^{\top}\) with orthonormal eigenvectors \(\mathbf{u}_i\) and eigenvalues \(\lambda_i > 0\). The ellipsoid’s axes point along the \(\mathbf{u}_i\), and its semi-axis in direction \(i\) has length proportional to \(\sqrt{\lambda_i}\). Take
The contours are ellipses tilted at 45°, elongated by \(\sqrt{1.8/0.2} = 3\), three times wider along the diagonal than across it. Strong positive correlation is that elongation.
Two structural facts make the family tractable. Marginals and conditionals of a Gaussian are Gaussian, with the conditional mean \(\boldsymbol{\mu}_1 + \boldsymbol{\Sigma}_{12}\boldsymbol{\Sigma}_{22}^{-1}(\mathbf{x}_2-\boldsymbol{\mu}_2)\), linear in what you conditioned on, which is where linear regression and Kalman filters come from. And any affine map of a Gaussian is Gaussian: with \(\boldsymbol{\Sigma}=\mathbf{L}\mathbf{L}^{\top}\), sampling is \(\mathbf{x} = \boldsymbol{\mu}+\mathbf{L}\boldsymbol{\epsilon}\) with \(\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\), the reparameterisation trick that makes VAEs differentiable.
Where this goes next
Why sums of almost anything drift towards the Gaussian is limit theorems; fitting these families to data is the subject of statistics basics.
Recap
- Choose by data type: binary → Bernoulli, counts → Poisson, waiting times → Exponential, probabilities → Beta/Dirichlet.
- Poisson has mean equal to variance, both \(\lambda\); Binomial\((n,\lambda/n)\) converges to it.
- Beta and Dirichlet are conjugate to Bernoulli and Categorical, with parameters acting as pseudo-counts.
- Gaussian level sets are ellipsoids with axes \(\mathbf{u}_i\) and widths \(\propto\sqrt{\lambda_i}\); \(\rho=0.8\) gives an axis ratio of 3.
- Zero correlation implies independence only under joint Gaussianity.
References
- Bishop, C. M. Pattern Recognition and Machine Learning, ch. 2. Springer, 2006.
- Murphy, K. P. Probabilistic Machine Learning: An Introduction. MIT Press, 2022.
- Kingma, D. P. & Welling, M. Auto-Encoding Variational Bayes. ICLR 2014.
- Blei, D. M., Ng, A. Y. & Jordan, M. I. Latent Dirichlet Allocation. JMLR 3, 993–1022, 2003.