Bayesian vs Frequentist: Two Meanings of the Word Probability
Published:
The disagreement
To a frequentist, \(\mathbb{P}(A)\) is the limiting relative frequency of \(A\) in repeated trials. A coin’s bias \(\theta\) is a fixed physical constant, not the outcome of a trial, so “\(\mathbb{P}(\theta > 0.5)\)” is not a meaningful expression — it is either true or false. Randomness lives entirely in the sampling, which is why every frequentist guarantee is a statement about what would happen across hypothetical repeated experiments.
To a Bayesian, \(\mathbb{P}(A)\) is a degree of belief obeying the probability axioms. Uncertainty about a fixed constant is still uncertainty, so \(\theta\) gets a distribution — before the data (the prior) and after (the posterior). The data, once observed, are not random at all; they are what you conditioned on.
Bayes’ theorem as inference
The denominator does not depend on \(\theta\), so for inference it is just the constant that makes the posterior integrate to one, and the working form is \(p(\theta\mid x) \propto p(x\mid\theta)\,p(\theta)\). The likelihood is the same object maximum likelihood maximises; the difference is that Bayes multiplies it by a prior and keeps the whole resulting function rather than its peak.
Beta–Binomial, worked
A conjugate prior is one that leaves the posterior in the same family, so the update is arithmetic on parameters rather than an integral. For binomial data the conjugate prior is the Beta.
Take \(\theta \sim \mathrm{Beta}(2,2)\) — symmetric, gently favouring values near \(0.5\), worth as much as two prior successes and two prior failures. Observe \(s = 7\) successes in \(n = 10\) trials. Since \(p(\theta) \propto \theta^{1}(1-\theta)^{1}\) and \(p(x\mid\theta)\propto\theta^{7}(1-\theta)^{3}\), the product is \(\theta^{8}(1-\theta)^{4}\):
The posterior mean is \(9/14 = 0.643\), sitting between the prior mean \(0.5\) and the MLE \(0.7\). It is exactly a weighted average, with weights given by the counts:
The prior carries \(\alpha+\beta = 4\) pseudo-observations against the data’s 10, so it holds 29% of the weight. Double the data to 20 trials and the prior’s share falls to 17%; the prior is a fixed amount of evidence that real data eventually swamp.
Credible versus confidence
The 95% equal-tailed credible interval for \(\mathrm{Beta}(9,5)\) runs from 0.386 to 0.861, and it means what people wish a confidence interval meant: given this prior and this data, there is a 95% probability that \(\theta\) lies in that range. You can also read off \(\mathbb{P}(\theta > 0.5 \mid x) = 0.867\) directly, a question a p-value cannot answer.
| Method | 95% interval | What it claims |
|---|---|---|
| Credible, \(\mathrm{Beta}(9,5)\) | [0.386, 0.861] | 95% posterior probability \(\theta\) is inside |
| Clopper–Pearson (exact) | [0.348, 0.933] | ≥95% of such intervals cover \(\theta\) |
| Wilson score | [0.397, 0.892] | ≈95% coverage |
| Wald, \(\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n}\) | [0.416, 0.984] | nominally 95%, actually much less |
When it matters and when it does not
With a large sample and a prior that is not sharply informative, the Bernstein–von Mises theorem says the posterior converges to a Gaussian centred at the MLE with variance given by the inverse Fisher information — the same object the frequentist asymptotics produce. At \(n = 10\) the credible and Clopper–Pearson intervals above differ by a lot; at \(n = 1000\) they would agree to three digits. The choice is then a matter of interpretation, not of numbers.
The genuine divergences are: small samples, where the prior does real work; problems with sequential or optional stopping, where frequentist error rates depend on the stopping rule and the posterior does not; hierarchical models, where partial pooling is natural in one framework and awkward in the other; and any setting where you want a direct probability about a hypothesis rather than a tail probability under a null.
Recap
- Frequentist: parameter fixed, data random, probability = long-run frequency. Bayesian: data given, parameter uncertain, probability = degree of belief.
- Posterior ∝ likelihood × prior; conjugacy makes the update arithmetic — Beta(α,β) with \(s\) of \(n\) becomes Beta(α+s, β+n−s).
- Beta(2,2) with 7/10 gives Beta(9,5): posterior mean 0.643, a weighted average of the prior mean 0.5 and the MLE 0.7 with weights 4/14 and 10/14.
- Credible interval [0.386, 0.861] claims 95% posterior probability; a confidence interval claims 95% coverage of the procedure. Different claims.
- Bernstein–von Mises: with enough data and a non-degenerate prior the two answers converge. Disagreement is a small-sample, stopping-rule or hierarchical phenomenon.
References
- Brown, L. D., Cai, T. T., & DasGupta, A. Interval Estimation for a Binomial Proportion. Statistical Science, 16(2), 101–133, 2001.
- Kass, R. E., & Wasserman, L. The Selection of Prior Distributions by Formal Rules. Journal of the American Statistical Association, 91(435), 1343–1370, 1996.
- van der Vaart, A. W. Asymptotic Statistics. Cambridge University Press, 1998.
- Efron, B., & Hastie, T. Computer Age Statistical Inference. Cambridge University Press, 2016.