Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Confidence Intervals: The Interval Is Random, the Parameter Is Not

6 minute read

Published:

TL;DR: Build a confidence interval by inverting a pivot: find a quantity whose distribution you know regardless of the unknown parameter, bracket it between two quantiles, and rearrange. The 95% refers to the long-run success rate of the procedure across repeated samples — the endpoints are the random objects, the parameter is a fixed unknown constant. Width shrinks as \(1/\sqrt{n}\), so buying half the uncertainty costs four times the data.

The sampling distribution and the standard error

Compute \(\bar{x}\) from a sample of size \(n\). Collect another sample and you get a different \(\bar{x}\). The distribution of those values across hypothetical repetitions is the sampling distribution of the mean, and its standard deviation is the standard error:

\[ \mathrm{SE}(\bar{x}) = \frac{\sigma}{\sqrt{n}} \;\approx\; \frac{s}{\sqrt{n}} \]

Two things separate the standard error from the sample standard deviation \(s\), and conflating them is common. \(s\) estimates the spread of individual observations and converges to \(\sigma\), a fixed property of the population that does not shrink with more data. The standard error measures the spread of the estimate and does shrink, at rate \(1/\sqrt{n}\).

Constructing the interval

For Gaussian data the standardised mean has a known distribution that does not depend on \(\mu\) or \(\sigma\) — a pivot:

\[ T = \frac{\bar{x} - \mu}{s/\sqrt{n}} \;\sim\; t_{n-1} \]

Let \(t^\star\) be the 0.975 quantile of \(t_{n-1}\). Then \(\mathbb{P}(-t^\star < T < t^\star) = 0.95\) by construction, and rearranging the inequality to isolate \(\mu\) gives

\[ \bar{x} - t^\star\frac{s}{\sqrt{n}} \;<\; \mu \;<\; \bar{x} + t^\star\frac{s}{\sqrt{n}} \]

The probability statement was made before the data arrived, about the random variables \(\bar{x}\) and \(s\). Rearranging an inequality does not move randomness from one side to the other.

Worked example. A sample of \(n = 25\) response times gives \(\bar{x} = 100\) ms and \(s = 15\) ms. Then \(\mathrm{SE} = 15/\sqrt{25} = 3.0\) ms, and \(t^\star_{24} = 2.0639\), so the half-width is \(2.0639 \times 3.0 = 6.19\) and the interval is \([93.81,\ 106.19]\).

What the 95% attaches to

Interview trap — the misinterpretation: "there is a 95% probability that \(\mu\) lies in [93.81, 106.19]" is wrong. Once the data are in, the endpoints are two specific numbers and \(\mu\) is a specific unknown constant; either it is in there or it is not, and there is no randomness left to carry a probability. What is 95% is the coverage of the procedure: if you repeated the whole experiment many times, 95% of the intervals so constructed would contain \(\mu\). The interval is random; the parameter is not. If you want "95% probability that the parameter is in this range", you want a credible interval, which requires a prior.
Twenty simulated 95% confidence intervals against a fixed true meanTwenty horizontal intervals, each from a simulated sample of 25 draws from a Normal distribution with mean 100 and standard deviation 15. A dashed vertical line marks the true mean of 100. Eighteen intervals, drawn in teal, cross the line. Two, drawn in orange — the seventeenth and the twentieth — fall entirely to one side and miss it. The true mean never moves; only the intervals do. 9095100105110 true μ = 100 (fixed) 2 of 20 miss
Notice which object moves. The dashed line is fixed; the intervals jump around it. Two of these twenty miss — over 20,000 replications of the same simulation the coverage was 94.8%, as advertised.

\(t\) or \(z\)

Use \(z = 1.96\) when \(\sigma\) is known; use \(t_{n-1}\) when you estimated it from the same data. Estimating \(\sigma\) injects extra variability — sometimes \(s\) comes out too small, which would make a \(z\)-interval too narrow — and the heavier tails of \(t\) pay for exactly that.

\(n\)multiplierhalf-width (\(s=15\))interval
25 (\(t_{24}\))2.06396.19[93.81, 106.19]
25 (\(z\), wrong here)1.96005.88[94.12, 105.88]
100 (\(t_{99}\))1.98422.98[97.02, 102.98]

At \(n=25\) the correction is about 5%; by \(n=100\) it is under 2% and the distinction stops mattering in practice. What does not stop mattering is the assumption behind the pivot: \(T\) is exactly \(t_{n-1}\) only for Gaussian data. For other distributions the interval is justified by the central limit theorem, which needs \(n\) large enough for \(\bar{x}\) to be near-Gaussian — and heavy tails or strong skew can make “large enough” run into the hundreds. When in doubt, use the bootstrap.

The \(1/\sqrt{n}\) tax

Half-width is \(t^\star s/\sqrt{n}\), so it falls as \(1/\sqrt{n}\). Going from \(n=25\) to \(n=100\) took the half-width from 6.19 to 2.98 — a factor of 2.08, the extra fraction coming from \(t^\star\) shrinking towards 1.96 as well. Quadrupling data halves uncertainty; a further tenfold reduction would need 10,000 samples.

Key Insight — why the square root, and what it costs: \(\mathrm{Var}(\bar x) = \sigma^2/n\) because variances of independent terms add while the \(1/n\) scaling squares, so the standard deviation keeps only \(1/\sqrt{n}\). The consequence for reported model accuracies is uncomfortable. Correctness on one example is Bernoulli, so at 90% accuracy the per-example standard deviation is \(\sqrt{0.9 \times 0.1} = 0.3\). Getting a 95% interval of ±1 percentage point therefore needs \(n = (1.96 \times 0.3 / 0.01)^2 \approx 3{,}460\) test examples, and ±0.5 points needs four times that, about 13,800. A great many published gaps are smaller than the interval of the test set they were measured on.

Recap

  • Standard error \(s/\sqrt{n}\) is the spread of the estimate; \(s\) is the spread of the observations and does not shrink with \(n\).
  • A CI comes from inverting a pivot — a quantity whose distribution is free of the unknown parameter.
  • 95% is the coverage of the procedure over repeated samples. The endpoints are random; \(\mu\) is a fixed constant.
  • Use \(t_{n-1}\) when \(\sigma\) is estimated: 2.064 at \(n=25\) versus 1.96, a 5% widening that fades by \(n \approx 100\).
  • Width scales as \(1/\sqrt{n}\): four times the data for half the width.

References

  1. Neyman, J. Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability. Philosophical Transactions of the Royal Society A, 236, 333–380, 1937.
  2. Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31, 337–350, 2016.
  3. Brown, L. D., Cai, T. T., & DasGupta, A. Interval Estimation for a Binomial Proportion. Statistical Science, 16(2), 101–133, 2001.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.