The Ten Distributions You Need, and the Geometry of the Multivariate Gaussian
Published:
The reference table
$\operatorname{Var}$ is the variance of a single draw; parameters are on the left.
| Distribution | Models | Parameters | Mean | Variance |
|---|---|---|---|---|
| Bernoulli$(p)$ | one binary trial | $p\in[0,1]$ | $p$ | $p(1-p)$ |
| Binomial$(n,p)$ | successes in $n$ trials | $n\in\mathbb{N},\,p$ | $np$ | $np(1-p)$ |
| Categorical$(\boldsymbol{\pi})$ | one draw from $K$ classes | $\boldsymbol{\pi}$, $\sum_k\pi_k=1$ | $\boldsymbol{\pi}$ | $\operatorname{diag}(\boldsymbol{\pi})-\boldsymbol{\pi}\boldsymbol{\pi}^{\top}$ |
| Poisson$(\lambda)$ | counts of rare independent events | $\lambda>0$ | $\lambda$ | $\lambda$ |
| Uniform$(a,b)$ | no preference on an interval | $a \lt b$ | $\frac{a+b}{2}$ | $\frac{(b-a)^2}{12}$ |
| Gaussian$(\mu,\sigma^2)$ | sums of many small effects | $\mu,\ \sigma^2>0$ | $\mu$ | $\sigma^2$ |
| Exponential$(\lambda)$ | memoryless waiting time | $\lambda>0$ | $1/\lambda$ | $1/\lambda^2$ |
| Beta$(\alpha,\beta)$ | an unknown probability | $\alpha,\beta>0$ | $\frac{\alpha}{\alpha+\beta}$ | $\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$ |
| Dirichlet$(\boldsymbol{\alpha})$ | an unknown probability vector | $\alpha_k>0$, $\alpha_0=\sum_k\alpha_k$ | $\alpha_k/\alpha_0$ | $\frac{\alpha_k(\alpha_0-\alpha_k)}{\alpha_0^2(\alpha_0+1)}$ |
| Laplace$(\mu,b)$ | sharply peaked, heavy-tailed noise | $\mu,\ b>0$ | $\mu$ | $2b^2$ |
Three relationships tie the table together. Binomial is a sum of independent Bernoullis. Poisson is the limit of Binomial$(n,\lambda/n)$ as $n\to\infty$, which is why it models rare events among many opportunities. Beta is the one-dimensional case of Dirichlet.
The three that matter most in ML
Categorical. Every classifier’s output layer. A softmax over $K$ logits is a categorical parameter vector, and the training loss is the negative log-likelihood of the observed class under it. Its covariance $\operatorname{diag}(\boldsymbol{\pi})-\boldsymbol{\pi}\boldsymbol{\pi}^{\top}$ is exactly the Jacobian of the softmax, which is why the softmax–cross-entropy gradient collapses to $\hat{\mathbf{y}} - \mathbf{y}$.
Beta and Dirichlet. These are the conjugate priors for Bernoulli and Categorical: start with Beta$(\alpha,\beta)$, observe $s$ successes and $f$ failures, and the posterior is Beta$(\alpha+s,\beta+f)$. The parameters act as pseudo-counts, which is the cleanest way to read Laplace smoothing — adding 1 to every count is a Dirichlet$(\mathbf{1})$ prior. Dirichlet is also what latent Dirichlet allocation puts over topic proportions.
| Laplace. Its log-density is $- | x-\mu | /b$ up to a constant, so a Laplace prior on weights gives an $\ell_1$ penalty and a Gaussian prior gives $\ell_2$. Lasso versus ridge is a choice of noise model, not just a choice of penalty. |
The multivariate Gaussian
For $\mathbf{x}\in\mathbb{R}^d$ with mean $\boldsymbol{\mu}$ and positive-definite covariance $\boldsymbol{\Sigma}$:
Everything about the shape is in the quadratic form. The quantity $\Delta^2 = (\mathbf{x}-\boldsymbol{\mu})^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu})$ is the squared Mahalanobis distance, and the density is constant wherever $\Delta$ is constant — so the level sets are ellipsoids.
Diagonalise, $\boldsymbol{\Sigma} = \mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^{\top}$ with orthonormal eigenvectors \(\mathbf{u}_i\) and eigenvalues $\lambda_i > 0$. The ellipsoid’s axes point along the \(\mathbf{u}_i\), and its semi-axis in direction $i$ has length proportional to $\sqrt{\lambda_i}$. Take
The contours are ellipses tilted at 45°, elongated by $\sqrt{1.8/0.2} = 3$ — three times wider along the diagonal than across it. Strong positive correlation is that elongation.
Two structural facts make the family tractable. Marginals and conditionals of a Gaussian are Gaussian, with the conditional mean \(\boldsymbol{\mu}_1 + \boldsymbol{\Sigma}_{12}\boldsymbol{\Sigma}_{22}^{-1}(\mathbf{x}_2-\boldsymbol{\mu}_2)\) — linear in what you conditioned on, which is where linear regression and Kalman filters come from. And any affine map of a Gaussian is Gaussian: with $\boldsymbol{\Sigma}=\mathbf{L}\mathbf{L}^{\top}$, sampling is $\mathbf{x} = \boldsymbol{\mu}+\mathbf{L}\boldsymbol{\epsilon}$ with $\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ — the reparameterisation trick that makes VAEs differentiable.
Where this goes next
Why sums of almost anything drift towards the Gaussian is limit theorems; fitting these families to data is the subject of statistics basics.
Recap
- Choose by data type: binary → Bernoulli, counts → Poisson, waiting times → Exponential, probabilities → Beta/Dirichlet.
- Poisson has mean equal to variance, both \(\lambda\); Binomial\((n,\lambda/n)\) converges to it.
- Beta and Dirichlet are conjugate to Bernoulli and Categorical, with parameters acting as pseudo-counts.
- Gaussian level sets are ellipsoids with axes \(\mathbf{u}_i\) and widths \(\propto\sqrt{\lambda_i}\); \(\rho=0.8\) gives an axis ratio of 3.
- Zero correlation implies independence only under joint Gaussianity.
References
- Bishop, C. M. Pattern Recognition and Machine Learning, ch. 2. Springer, 2006.
- Murphy, K. P. Probabilistic Machine Learning: An Introduction. MIT Press, 2022.
- Kingma, D. P. & Welling, M. Auto-Encoding Variational Bayes. ICLR 2014.
- Blei, D. M., Ng, A. Y. & Jordan, M. I. Latent Dirichlet Allocation. JMLR 3, 993–1022, 2003.