Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Random Variables: A Density Is Not a Probability

5 minute read

Published:

TL;DR: A random variable is a function from outcomes to numbers. Discrete ones carry a PMF, which gives probabilities directly; continuous ones carry a PDF, which does not, \(p(x)\) is a density per unit \(x\), can exceed 1, and only becomes a probability after integration. The CDF works for both. Transforming a variable multiplies its density by a Jacobian determinant, and inverting the CDF turns uniform noise into samples from anything.

From outcomes to numbers

A random variable \(X\) is a measurable function \(X:\Omega\to\mathbb{R}\), it labels each outcome with a number. Rolling two dice, \(\Omega\) has 36 elements, and \(X =\) “sum” collapses them onto \(\{2,\dots,12\}\). Probability then transfers: \(P(X = 7) = P(\{\omega : X(\omega)=7\}) = 6/36\). This pushforward is why we can stop mentioning \(\Omega\) almost immediately; all the information lives in the induced distribution on \(\mathbb{R}\).

Discrete variables take countably many values and are described by a PMF \(p(x) = P(X=x)\), with \(p(x)\in[0,1]\) and \(\sum_x p(x)=1\). Each value is a genuine probability.

Continuous variables take uncountably many values, and each single value has probability zero. What exists instead is a PDF \(p(x)\ge 0\) with \(\int p(x)\,dx = 1\), defined so that

\[ P(a \le X \le b) = \int_a^b p(x)\,dx . \]
Interview trap, "\(p(x)\) is the probability of \(x\)": it is not, and it is not even bounded by 1. A uniform distribution on \([0, 0.5]\) has density \(p(x) = 2\) everywhere on that interval, because \(2 \times 0.5 = 1\). A narrow Gaussian with \(\sigma = 0.01\) peaks at \(1/(\sigma\sqrt{2\pi}) \approx 39.9\). The density has units of probability per unit \(x\); only \(p(x)\,dx\) is dimensionless. This is also why log-likelihoods of continuous models can be positive, and why comparing them across different data scalings is meaningless.

The object that unifies both cases is the CDF

\[ F(x) = P(X \le x), \]

which is non-decreasing, right-continuous, and runs from 0 to 1. It jumps at atoms of a discrete variable and is continuous with \(F'(x) = p(x)\) for a continuous one. Because \(F\) always exists, statements phrased through CDFs, convergence in distribution, quantiles, stochastic dominance, apply uniformly to both kinds.

Change of variables

Push \(X\) through a differentiable, strictly monotone map \(Y = g(X)\). Probability mass is conserved, but it gets stretched or compressed, so the density must be rescaled by how much \(g\) stretches the axis:

\[ p_Y(y) = p_X\!\left(g^{-1}(y)\right)\left|\frac{d}{dy}g^{-1}(y)\right| . \]

In \(d\) dimensions the derivative becomes the Jacobian matrix and the stretch factor becomes the absolute determinant, \(p_Y(\mathbf{y}) = p_X(g^{-1}(\mathbf{y}))\,\bigl\lvert \det J_{g^{-1}}(\mathbf{y})\bigr\rvert\), because the determinant is exactly the local volume-scaling factor of a linear map.

A concrete case: let \(X\sim\mathrm{Uniform}(0,1)\) and \(Y = -\log X\). Then \(g^{-1}(y) = e^{-y}\) with derivative \(-e^{-y}\), and \(p_X = 1\) on the unit interval, so

\[ p_Y(y) = 1 \cdot \left| -e^{-y}\right| = e^{-y}, \qquad y > 0, \]

the standard exponential. Uniform noise became an exponential purely through the Jacobian.

This identity is the whole content of normalising flows: compose invertible maps, track \(\sum \log\lvert \det J\rvert\), and you have an exact log-likelihood for the transformed density. Architectures like coupling layers exist precisely because a general \(\det J\) costs \(\mathcal{O}(d^3)\), whereas a triangular Jacobian costs \(\mathcal{O}(d)\).

Quantiles and inverse-transform sampling

The quantile function is the generalised inverse of the CDF, \(F^{-1}(u) = \inf\{x : F(x)\ge u\}\) for \(u\in(0,1)\). The infimum handles flat regions and jumps, so \(F^{-1}\) is defined even when \(F\) is not strictly increasing. \(F^{-1}(0.5)\) is the median.

Its consequence is one of the most useful facts in simulation: if \(U\sim\mathrm{Uniform}(0,1)\) then \(F^{-1}(U)\) has CDF \(F\). The proof is one line, \(P(F^{-1}(U)\le x) = P(U \le F(x)) = F(x)\), using only that \(U\) is uniform. Every sampler for a univariate distribution can therefore be built from a uniform random number generator, provided you can invert \(F\).

For the exponential with rate \(\lambda\), \(F(x) = 1-e^{-\lambda x}\), so \(F^{-1}(u) = -\log(1-u)/\lambda\). With \(\lambda = 2\) and a draw \(u = 0.7\): \(x = -\log(0.3)/2 = 1.20397/2 = 0.60199\). Since \(1-U\) is also uniform, implementations usually write \(-\log(U)/\lambda\).

Inverse-transform sampling from an exponential CDFA plot of the exponential CDF with rate 2, rising from 0 towards 1. A uniform draw of 0.7 on the vertical axis is carried horizontally to the curve and then dropped vertically to the horizontal axis, landing at x equals 0.602. A second draw of 0.3 lands at x equals 0.178. The flat upper part of the curve maps a broad span of u values into a narrow span of large x values, and vice versa. u=0.7u=0.3 1.00 0.6020.178 x F(x) = 1 − e^(−2x) F(x)
Notice that where the CDF is steep, a small slice of \(u\) maps to a small slice of \(x\), that is where the density is high. Steepness of \(F\) is the density, which is why inverting it lands samples in the right places automatically.
Key Insight, the density is a bookkeeping device, the CDF is the real object: the CDF exists for every random variable, discrete, continuous or mixed, and determines the distribution completely. The PDF exists only when \(F\) is differentiable, and its value at a point is meaningless in isolation. When a claim about densities looks paradoxical, a density above 1, a likelihood that changes when you switch from metres to centimetres, restate it in terms of \(F\) and the paradox disappears.

Where this goes next

Summarising a distribution by a few numbers rather than a whole function is expectation and variance; the standard families and their densities are catalogued in common distributions.

Recap

  • A random variable is a measurable map \(\Omega\to\mathbb{R}\); the distribution is the pushforward of \(P\).
  • A PMF gives probabilities; a PDF gives density per unit \(x\) and may exceed 1. Only \(\int p\,dx\) is a probability.
  • The CDF \(F(x)=P(X\le x)\) exists always and characterises the distribution.
  • Under \(Y=g(X)\), densities pick up \(|\det J|\), the identity behind normalising flows.
  • \(F^{-1}(U)\) with \(U\sim\mathrm{Uniform}(0,1)\) samples from \(F\); for \(\mathrm{Exp}(\lambda)\) that is \(-\log(U)/\lambda\).

References

  1. Devroye, L. Non-Uniform Random Variate Generation. Springer, 1986.
  2. Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S. & Lakshminarayanan, B. Normalizing Flows for Probabilistic Modeling and Inference. JMLR 22(57), 2021.
  3. Rezende, D. J. & Mohamed, S. Variational Inference with Normalizing Flows. ICML 2015.
  4. Casella, G. & Berger, R. L. Statistical Inference, 2nd ed. Duxbury, 2002.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.