Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Expectation and Variance: Linearity Is Free, Additivity Is Not

5 minute read

Published:

TL;DR: \(\mathbb{E}[X+Y]=\mathbb{E}X+\mathbb{E}Y\) always, with no independence assumption — which makes indicator decompositions absurdly effective. Variance only adds when the covariance vanishes: \(\operatorname{Var}(X+Y)=\operatorname{Var}X+\operatorname{Var}Y+2\operatorname{Cov}(X,Y)\). Correlation is normalised covariance and detects linear association only, so zero correlation does not mean independence. The laws of total expectation and total variance let you compute either by conditioning.

Expectation

For a discrete variable $\mathbb{E}[X]=\sum_x x\,p(x)$; for a continuous one $\mathbb{E}[X]=\int x\,p(x)\,dx$. Both are the same object — an integral against the distribution — and both can fail to exist when the tails are heavy enough, as for the Cauchy.

The property that matters is linearity:

\[ \mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y] \]

for any $a,b$ and any joint distribution of $X$ and $Y$. Dependence is irrelevant because the identity is a statement about the integral, which is linear before any probabilistic structure is imposed.

That freedom is what makes indicator decompositions so effective. Take a uniformly random permutation of $n$ items and count fixed points — elements left in their own position. Let $I_i = 1$ if item $i$ is fixed. The $I_i$ are heavily dependent, but $\mathbb{E}[I_i] = P(\text{item } i \text{ fixed}) = 1/n$, so

\[ \mathbb{E}\left[\textstyle\sum_{i=1}^{n} I_i\right] = n \cdot \frac{1}{n} = 1 \]

for every $n$. No inclusion–exclusion, no derangement formula. Whenever a quantity is a count, write it as a sum of indicators before doing anything else.

Interview trap — assuming linearity needs independence: it does not. The common follow-up, "what about \(\mathbb{E}[XY]\)?", does: \(\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]\) requires uncorrelatedness (which independence implies). Keep the two straight — sums are always free, products are not.

Variance and the covariance term

Variance is the expected squared deviation, $\operatorname{Var}(X)=\mathbb{E}[(X-\mathbb{E}X)^2] = \mathbb{E}[X^2]-(\mathbb{E}X)^2$. It is not linear: $\operatorname{Var}(aX)=a^2\operatorname{Var}(X)$, and constants shift the mean without moving the spread.

Expanding the square of a sum leaves a cross term:

\[ \operatorname{Var}(X+Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X,Y), \qquad \operatorname{Cov}(X,Y) = \mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]. \]

The cross term is why averaging $n$ model predictions reduces variance by a factor of $n$ only when their errors are uncorrelated. For an equally weighted ensemble with common variance $\sigma^2$ and pairwise correlation $\rho$, the variance of the average is \(\sigma^2\!\left[\rho + (1-\rho)/n\right]\), which floors at $\rho\sigma^2$ however many members you add. Ensembling buys nothing beyond that floor, which is why diversity, not count, is the design objective.

Correlation normalises covariance onto $[-1,1]$:

\[ \rho(X,Y) = \frac{\operatorname{Cov}(X,Y)}{\sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)}} . \]
The bound is Cauchy–Schwarz, and $\rho=1$ exactly when $Y$ is an affine function of $X$. Correlation is a measure of linear association, nothing more.

Uncorrelated does not mean independent

Let $X$ be uniform on \(\{-1,0,1\}\) and set $Y = X^2$. Then $\mathbb{E}[X]=0$ and $\mathbb{E}[XY]=\mathbb{E}[X^3] = \tfrac13(-1+0+1) = 0$, so

\[ \operatorname{Cov}(X,Y) = 0 - 0 \cdot \mathbb{E}[Y] = 0, \qquad \rho = 0 . \]

Yet $Y$ is a deterministic function of $X$. Concretely, $P(Y=0)=\tfrac13$ while $P(Y=0\mid X=1)=0$, so the two are as dependent as it is possible to be. The same works for $X\sim\mathcal{N}(0,1)$, $Y=X^2$, since $\mathbb{E}[X^3]=0$ by symmetry.

The converse direction does hold: independence implies $\mathbb{E}[XY]=\mathbb{E}X\,\mathbb{E}Y$, hence zero covariance. The one place the implication reverses is the jointly Gaussian case — for a multivariate normal, zero covariance really does mean independence, which is why the trap is so easy to fall into after a term of working with Gaussians.

Zero correlation with total dependenceA scatter of points lying exactly on the parabola y equals x squared, symmetric about the vertical axis, for x from minus one to one. A dashed horizontal best-fit line has slope zero, indicating a correlation of zero, even though every point lies exactly on the curve so y is a deterministic function of x. best linear fit: slope 0 xy ρ = 0, yet y = x² exactly
Notice that the fitted line is flat: the symmetric parabola has no linear component at all. Correlation measures the slope of the best straight line, so it reports zero while the relationship is deterministic.

Conditioning: total expectation and total variance

Conditioning gives two decompositions that are used constantly. The law of total expectation:

\[ \mathbb{E}[Y] = \mathbb{E}\bigl[\,\mathbb{E}[Y \mid X]\,\bigr] \]

— average within groups, then average the group averages, weighted by group size. The law of total variance splits spread the same way:

\[ \operatorname{Var}(Y) = \underbrace{\mathbb{E}\bigl[\operatorname{Var}(Y\mid X)\bigr]}_{\text{within-group}} + \underbrace{\operatorname{Var}\bigl(\mathbb{E}[Y\mid X]\bigr)}_{\text{between-group}} . \]

A worked case: a value comes from one of two groups with equal probability; group 0 has mean 0 and variance 1, group 1 has mean 4 and variance 1. Within-group term: $\mathbb{E}[\operatorname{Var}(Y\mid X)] = 1$. Between-group term: the conditional means are 0 and 4 with probability $\tfrac12$ each, so their variance is $8 - 2^2 = 4$. Total: $\operatorname{Var}(Y) = 1 + 4 = 5$. Checking directly, $\mathbb{E}[Y]=2$ and $\mathbb{E}[Y^2] = \tfrac12(1+0)+\tfrac12(1+16) = 9$, giving $9-4=5$.

Key Insight — total variance is the uncertainty decomposition: read \(X\) as "which model" in a deep ensemble and \(Y\) as the prediction. The within-group term \(\mathbb{E}[\operatorname{Var}(Y\mid X)]\) is aleatoric uncertainty — noise no model can remove — and the between-group term \(\operatorname{Var}(\mathbb{E}[Y\mid X])\) is epistemic, the disagreement between models, which more data can shrink. The split reported by Bayesian deep learning methods is precisely this identity.

Where this goes next

Means and variances for the standard families are tabulated in common distributions; what happens to a sample mean as $n$ grows is limit theorems.

Recap

  • Linearity of expectation holds for any dependence; \(\mathbb{E}[XY]=\mathbb{E}X\,\mathbb{E}Y\) does not.
  • \(\operatorname{Var}(X+Y)=\operatorname{Var}X+\operatorname{Var}Y+2\operatorname{Cov}(X,Y)\); ensemble averaging floors at \(\rho\sigma^2\).
  • Correlation captures linear association only: \(X\) uniform on \(\{-1,0,1\}\) with \(Y=X^2\) has \(\rho=0\) and total dependence.
  • Independence \(\Rightarrow\) uncorrelated, never the reverse — except for jointly Gaussian variables.
  • Total variance splits into within-group (aleatoric) and between-group (epistemic) parts.

References

  1. Ross, S. M. A First Course in Probability, 10th ed. Pearson, 2018.
  2. Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer, 2004.
  3. Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NeurIPS 2017.
  4. Kendall, A. & Gal, Y. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? NeurIPS 2017.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.