Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Expectation and Variance: Linearity Is Free, Additivity Is Not

5 minute read

Published:

TL;DR: \(\mathbb{E}[X+Y]=\mathbb{E}X+\mathbb{E}Y\) always, with no independence assumption, which makes indicator decompositions absurdly effective. Variance only adds when the covariance vanishes: \(\operatorname{Var}(X+Y)=\operatorname{Var}X+\operatorname{Var}Y+2\operatorname{Cov}(X,Y)\). Correlation is normalised covariance and detects linear association only, so zero correlation does not mean independence. The laws of total expectation and total variance let you compute either by conditioning.

Expectation

For a discrete variable \(\mathbb{E}[X]=\sum_x x\,p(x)\); for a continuous one \(\mathbb{E}[X]=\int x\,p(x)\,dx\). Both are the same object, an integral against the distribution, and both can fail to exist when the tails are heavy enough, as for the Cauchy.

The property that matters is linearity:

\[ \mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y] \]

for any \(a,b\) and any joint distribution of \(X\) and \(Y\). Dependence is irrelevant because the identity is a statement about the integral, which is linear before any probabilistic structure is imposed.

That freedom is what makes indicator decompositions so effective. Take a uniformly random permutation of \(n\) items and count fixed points, elements left in their own position. Let \(I_i = 1\) if item \(i\) is fixed. The \(I_i\) are heavily dependent, but \(\mathbb{E}[I_i] = P(\text{item } i \text{ fixed}) = 1/n\), so

\[ \mathbb{E}\left[\textstyle\sum_{i=1}^{n} I_i\right] = n \cdot \frac{1}{n} = 1 \]

for every \(n\). No inclusion–exclusion, no derangement formula. Whenever a quantity is a count, write it as a sum of indicators before doing anything else.

Interview trap, assuming linearity needs independence: it does not. The common follow-up, "what about \(\mathbb{E}[XY]\)?", does: \(\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]\) requires uncorrelatedness (which independence implies). Keep the two straight, sums are always free, products are not.

Variance and the covariance term

Variance is the expected squared deviation, \(\operatorname{Var}(X)=\mathbb{E}[(X-\mathbb{E}X)^2] = \mathbb{E}[X^2]-(\mathbb{E}X)^2\). It is not linear: \(\operatorname{Var}(aX)=a^2\operatorname{Var}(X)\), and constants shift the mean without moving the spread.

Expanding the square of a sum leaves a cross term:

\[ \operatorname{Var}(X+Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X,Y), \qquad \operatorname{Cov}(X,Y) = \mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]. \]

The cross term is why averaging \(n\) model predictions reduces variance by a factor of \(n\) only when their errors are uncorrelated. For an equally weighted ensemble with common variance \(\sigma^2\) and pairwise correlation \(\rho\), the variance of the average is \(\sigma^2\!\left[\rho + (1-\rho)/n\right]\), which floors at \(\rho\sigma^2\) however many members you add. Ensembling buys nothing beyond that floor, which is why diversity, not count, is the design objective.

Correlation normalises covariance onto \([-1,1]\):

\[ \rho(X,Y) = \frac{\operatorname{Cov}(X,Y)}{\sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)}} . \]

The bound is Cauchy–Schwarz, and \(\lvert \rho\rvert=1\) exactly when \(Y\) is an affine function of \(X\). Correlation is a measure of linear association, nothing more.

Uncorrelated does not mean independent

Let \(X\) be uniform on \(\{-1,0,1\}\) and set \(Y = X^2\). Then \(\mathbb{E}[X]=0\) and \(\mathbb{E}[XY]=\mathbb{E}[X^3] = \tfrac13(-1+0+1) = 0\), so

\[ \operatorname{Cov}(X,Y) = 0 - 0 \cdot \mathbb{E}[Y] = 0, \qquad \rho = 0 . \]

Yet \(Y\) is a deterministic function of \(X\). Concretely, \(P(Y=0)=\tfrac13\) while \(P(Y=0\mid X=1)=0\), so the two are as dependent as it is possible to be. The same works for \(X\sim\mathcal{N}(0,1)\), \(Y=X^2\), since \(\mathbb{E}[X^3]=0\) by symmetry.

The converse direction does hold: independence implies \(\mathbb{E}[XY]=\mathbb{E}X\,\mathbb{E}Y\), hence zero covariance. The one place the implication reverses is the jointly Gaussian case, for a multivariate normal, zero covariance really does mean independence, which is why the trap is so easy to fall into after a term of working with Gaussians.

Zero correlation with total dependenceA scatter of points lying exactly on the parabola y equals x squared, symmetric about the vertical axis, for x from minus one to one. A dashed horizontal best-fit line has slope zero, indicating a correlation of zero, even though every point lies exactly on the curve so y is a deterministic function of x. best linear fit: slope 0 xy ρ = 0, yet y = x² exactly
Notice that the fitted line is flat: the symmetric parabola has no linear component at all. Correlation measures the slope of the best straight line, so it reports zero while the relationship is deterministic.

Conditioning: total expectation and total variance

Conditioning gives two decompositions that are used constantly. The law of total expectation:

\[ \mathbb{E}[Y] = \mathbb{E}\bigl[\,\mathbb{E}[Y \mid X]\,\bigr] \]

, average within groups, then average the group averages, weighted by group size. The law of total variance splits spread the same way:

\[ \operatorname{Var}(Y) = \underbrace{\mathbb{E}\bigl[\operatorname{Var}(Y\mid X)\bigr]}_{\text{within-group}} + \underbrace{\operatorname{Var}\bigl(\mathbb{E}[Y\mid X]\bigr)}_{\text{between-group}} . \]

A worked case: a value comes from one of two groups with equal probability; group 0 has mean 0 and variance 1, group 1 has mean 4 and variance 1. Within-group term: \(\mathbb{E}[\operatorname{Var}(Y\mid X)] = 1\). Between-group term: the conditional means are 0 and 4 with probability \(\tfrac12\) each, so their variance is \(8 - 2^2 = 4\). Total: \(\operatorname{Var}(Y) = 1 + 4 = 5\). Checking directly, \(\mathbb{E}[Y]=2\) and \(\mathbb{E}[Y^2] = \tfrac12(1+0)+\tfrac12(1+16) = 9\), giving \(9-4=5\).

Key Insight, total variance is the uncertainty decomposition: read \(X\) as "which model" in a deep ensemble and \(Y\) as the prediction. The within-group term \(\mathbb{E}[\operatorname{Var}(Y\mid X)]\) is aleatoric uncertainty, noise no model can remove, and the between-group term \(\operatorname{Var}(\mathbb{E}[Y\mid X])\) is epistemic, the disagreement between models, which more data can shrink. The split reported by Bayesian deep learning methods is precisely this identity.

Where this goes next

Means and variances for the standard families are tabulated in common distributions; what happens to a sample mean as \(n\) grows is limit theorems.

Recap

  • Linearity of expectation holds for any dependence; \(\mathbb{E}[XY]=\mathbb{E}X\,\mathbb{E}Y\) does not.
  • \(\operatorname{Var}(X+Y)=\operatorname{Var}X+\operatorname{Var}Y+2\operatorname{Cov}(X,Y)\); ensemble averaging floors at \(\rho\sigma^2\).
  • Correlation captures linear association only: \(X\) uniform on \(\{-1,0,1\}\) with \(Y=X^2\) has \(\rho=0\) and total dependence.
  • Independence \(\Rightarrow\) uncorrelated, never the reverse, except for jointly Gaussian variables.
  • Total variance splits into within-group (aleatoric) and between-group (epistemic) parts.

References

  1. Ross, S. M. A First Course in Probability, 10th ed. Pearson, 2018.
  2. Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer, 2004.
  3. Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NeurIPS 2017.
  4. Kendall, A. & Gal, Y. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? NeurIPS 2017.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.