Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Hypothesis Testing: What a p-Value Actually Measures

7 minute read

Published:

TL;DR: Assume the null, compute a statistic, and ask how often the null would produce something at least this extreme. That tail probability is the p-value: \(\mathbb{P}(\text{data at least this extreme} \mid H_0)\), not \(\mathbb{P}(H_0 \mid \text{data})\). A small p-value with a tiny effect is a large sample, not a discovery; a large p-value with an underpowered study is no evidence of absence. And every extra test you run inflates the chance that at least one comes back "significant" by luck.

The machinery

A test needs three parts. The null hypothesis \(H_0\) is a specific claim precise enough to have a distribution — “the two means are equal” — and the alternative \(H_1\) is its complement, or a directional part of it. The test statistic is a function of the data whose distribution under \(H_0\) is known. You then locate your observed statistic in that null distribution and report how far into the tail it fell.

A two-sample test, all the arithmetic

Two variants of an onboarding flow, measuring time-on-task in seconds. Variant A: \(n_1 = 40\), \(\bar{x}_1 = 52.0\), \(s_1 = 12.0\). Variant B: \(n_2 = 40\), \(\bar{x}_2 = 57.0\), \(s_2 = 14.0\). Test \(H_0: \mu_1 = \mu_2\) against a two-sided alternative.

Pool the two variance estimates, weighting by degrees of freedom:

\[ s_p^2 = \frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1+n_2-2} = \frac{39(144) + 39(196)}{78} = \frac{13260}{78} = 170 \]

So \(s_p = \sqrt{170} = 13.038\). The standard error of the difference of means is

\[ \mathrm{SE} = s_p\sqrt{\tfrac{1}{n_1}+\tfrac{1}{n_2}} = 13.038\sqrt{0.05} = 2.9155 \]

and the statistic is \(t = (57.0 - 52.0)/2.9155 = 1.715\) on \(n_1+n_2-2 = 78\) degrees of freedom. The two-sided critical value is \(t^\star_{78} = 1.991\), so at \(\alpha = 0.05\) we do not reject. The exact two-sided p-value is 0.090: under the null, a gap of 5 seconds or more in either direction would arise about 9% of the time. The 95% interval for the difference is \(5 \pm 1.991 \times 2.9155 = [-0.80,\ 10.80]\) — it contains zero, which is the same conclusion in a more informative form.

The null distribution, the rejection region and the observed statisticA bell-shaped t distribution with 78 degrees of freedom, centred at zero. The two tails beyond plus and minus 1.991 are shaded orange and together hold 5 percent of the area — the rejection region at alpha 0.05. A vertical marker at t equals 1.715, the observed statistic, sits just inside the right-hand boundary, corresponding to a two-sided p-value of 0.090. observed t = 1.715 +1.991 −1.991 2.5% tail 2.5% tail −4−2024 t statistic under H₀ (78 degrees of freedom); two-sided p = 0.090
Notice how little separates the two conclusions. The observed statistic falls just short of the boundary, so a 0.05 threshold reports "no effect" for data that is also perfectly compatible with a 10-second slowdown. The interval carries information the verdict discards.
Interview trap — what a p-value is: it is \(\mathbb{P}(T \ge t_{\text{obs}} \mid H_0)\), the probability of data at least this extreme given the null is true. It is not the probability that the null is true, not the probability the result was chance, not one minus the probability of replication, and not a measure of effect size. Inverting the conditional requires a prior — see Bayesian versus frequentist. A related trap: \(p > 0.05\) does not mean the null is true. Our test above returned \(p = 0.090\) while remaining compatible with a real 10-second effect; failing to reject is not evidence of no difference.

Errors, and the one people forget

A Type I error rejects a true null; its rate is \(\alpha\), which you choose. A Type II error fails to reject a false null; its rate \(\beta\) you do not choose — it follows from the effect size, the noise and \(n\). Power is \(1-\beta\), the probability of detecting a real effect of a given size.

In the example above, if the true difference really is 5 seconds with \(\sigma \approx 13.0\), then 40 per group gives power of only about 0.40. The study was more likely to miss the effect than to find it. Reaching the conventional 80% would need roughly 107 per group — from \(n \approx 2(z_{0.975}+z_{0.80})^2\sigma^2/\delta^2 = 2(1.96+0.84)^2(170)/25\). Power is decided before the data are collected, and a non-significant result from an underpowered study carries almost no information.

Significant versus meaningful

Statistical significance says an effect is distinguishable from zero; it says nothing about size. Because the standard error shrinks as \(1/\sqrt{n}\), any non-zero effect becomes significant at large enough \(n\). With ten million users, a 0.01% conversion change is detectable and irrelevant. Always report the effect size and its interval, and decide separately whether the magnitude justifies the change.

Many tests at once

Testing \(m\) independent true nulls at \(\alpha = 0.05\), the chance of at least one false positive is \(1 - 0.95^m\): 40% at \(m=10\), 64% at \(m=20\), 99.4% at \(m=100\). Two corrections dominate.

Bonferroni tests each at \(\alpha/m\), bounding the probability of any false positive at \(\alpha\). At \(m=20\) that means a threshold of 0.0025 — safe, and often so conservative that real effects are lost.

Benjamini–Hochberg controls the false discovery rate, the expected fraction of rejections that are false. Sort the p-values \(p_{(1)} \le \dots \le p_{(m)}\), find the largest \(k\) with \(p_{(k)} \le \frac{k}{m}\alpha\), and reject the first \(k\). It is far more powerful when many alternatives are genuinely true, which is why it is standard in genomics and in large feature screens.

p-hacking is what happens when \(m\) is not counted. Trying several metrics, several subgroups, or several stopping points and reporting only the significant one gives a nominal 0.05 test whose real Type I rate is far higher — Simmons, Nelson and Simonsohn (2011) showed that a handful of ordinary, undisclosed analyst choices pushes it above 60%. The defence is to pre-register the primary metric and the sample size.

Key Insight — the test is a decision rule, not a measurement: Neyman and Pearson designed testing to control long-run error rates for repeated decisions, not to grade the evidence in one experiment. That is why "\(p = 0.049\) versus \(p = 0.051\)" is a meaningless distinction being made to look decisive, and why a confidence interval — which reports the estimate, its precision, and the null verdict together — is almost always the better thing to publish. A p-value throws away the effect size; the interval keeps it.

Recap

  • p-value = \(\mathbb{P}(\text{as extreme or more} \mid H_0)\). Never \(\mathbb{P}(H_0 \mid \text{data})\), and never an effect size.
  • Worked test: \(s_p^2 = 170\), SE = 2.9155, \(t = 1.715\) on 78 df, \(p = 0.090\), CI for the difference \([-0.80, 10.80]\).
  • \(\alpha\) is chosen; \(\beta\) is inherited. That study had ~40% power for the effect it was looking for; 80% would need ~107 per group.
  • Large \(n\) makes trivial effects significant — report magnitude and interval, not just the verdict.
  • Twenty tests at \(\alpha=0.05\) give a 64% chance of at least one false positive. Bonferroni controls any-false-positive; Benjamini–Hochberg controls the false discovery rate.

References

  1. Neyman, J., & Pearson, E. S. On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philosophical Transactions of the Royal Society A, 231, 289–337, 1933.
  2. Wasserstein, R. L., & Lazar, N. A. The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133, 2016.
  3. Benjamini, Y., & Hochberg, Y. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society B, 57(1), 289–300, 1995.
  4. Simmons, J. P., Nelson, L. D., & Simonsohn, U. False-Positive Psychology. Psychological Science, 22(11), 1359–1366, 2011.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.