Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Conditional Probability and Bayes: Why a 99% Accurate Test Means Almost Nothing

5 minute read

Published:

TL;DR: Conditioning restricts the sample space and renormalises: \(P(A\mid B) = P(A\cap B)/P(B)\). From that one definition come the chain rule, the law of total probability and Bayes' theorem. The famous failure mode is ignoring the prior: with a 1% base rate, a test with 99% sensitivity and 95% specificity gives a positive predictive value of exactly 1/6. Independence and conditional independence are logically unrelated, neither implies the other.

Conditioning is renormalisation

Learning that \(B\) occurred throws away every outcome outside \(B\) and rescales what remains so it sums to one:

\[ P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B) > 0 . \]

\(P(\cdot \mid B)\) is a genuine probability measure on the smaller space, it satisfies all three Kolmogorov axioms, so every identity you know still holds inside a conditional.

Rearranging gives the chain rule, \(P(A\cap B) = P(A\mid B)P(B)\), which iterates to any number of variables:

\[ P(x_1, \dots, x_n) = \prod_{t=1}^{n} P(x_t \mid x_1,\dots,x_{t-1}). \]

That factorisation is exact, requires no assumptions, and is the entire justification for autoregressive language models: predicting the next token given the prefix is not an approximation to the joint distribution, it is the joint distribution.

If \(\{B_1,\dots,B_k\}\) partitions \(\Omega\), the law of total probability reassembles a marginal from conditionals, \(P(A) = \sum_i P(A\mid B_i)P(B_i)\), averaging over cases, weighted by how often each case arises.

Bayes’ theorem

Write the joint two ways, \(P(A\mid B)P(B) = P(B\mid A)P(A)\), and solve:

\[ \underbrace{P(H \mid E)}_{\text{posterior}} = \frac{\overbrace{P(E \mid H)}^{\text{likelihood}}\ \overbrace{P(H)}^{\text{prior}}}{\underbrace{P(E)}_{\text{evidence}}}, \qquad P(E) = \sum_{h} P(E\mid h)P(h). \]

\(H\) is a hypothesis, \(E\) the evidence. The theorem is a one-line rearrangement; the difficulty is never the algebra, it is remembering that \(P(E\mid H)\) and \(P(H\mid E)\) are different numbers and that the prior \(P(H)\) has a vote.

The medical test, computed

A disease affects 1% of the screened population. The test has sensitivity \(P(+\mid D) = 0.99\) and specificity \(P(-\mid \neg D) = 0.95\), so its false-positive rate is \(P(+\mid \neg D) = 0.05\). You test positive.

Evidence first, by total probability:

\[ P(+) = 0.99 \times 0.01 + 0.05 \times 0.99 = 0.0099 + 0.0495 = 0.0594 . \]

Then Bayes:

\[ P(D \mid +) = \frac{0.0099}{0.0594} = \frac{1}{6} \approx 16.7\% . \]

Counting bodies makes it obvious. Screen 100,000 people:

GroupSizeTest positive
Diseased1,000990
Healthy99,0004,950
All positives 5,940

Of 5,940 positive results, 990 are genuine: \(990/5940 = 1/6\). The false positives outnumber the true ones five to one purely because the healthy group is 99 times larger. Raise the base rate to 10%, testing a symptomatic group rather than screening everyone, and the same test gives \(0.099/0.144 = 68.75\%\). The test did not change; the prior did.

Natural-frequency tree for the medical testA tree starting from 100,000 people screened. It splits into 1,000 diseased and 99,000 healthy. The diseased branch splits into 990 testing positive and 10 negative. The healthy branch splits into 4,950 testing positive and 94,050 negative. The two positive groups total 5,940, of which 990 are true positives, giving a positive predictive value of 990 divided by 5,940, which equals one sixth or 16.7 per cent. 100,000 1,000 diseased 99,000 healthy 990 test + 10 test − 4,950 test + 94,050 test − 5,940 positives 990 of them real = 16.7% sensitivity 99%, specificity 95%, prevalence 1%
Notice which box is large: 5% of 99,000 healthy people is 4,950 false positives, five times the 990 true ones. The test's error rate is small; the group it applies to is not.
Interview trap, base-rate neglect: asked this question, most people answer "about 99%", or "about 95%". Both confuse \(P(+\mid D)\) with \(P(D\mid +)\). Whenever a conditional probability is quoted, ask which way round it runs and what the base rate is. Presenting the problem in counts rather than percentages, as in the table above, reliably fixes the intuition, a finding replicated by Gigerenzer and Hoffrage (1995).

Independence is not conditional independence

\(A\) and \(B\) are independent when \(P(A\cap B) = P(A)P(B)\), equivalently \(P(A\mid B) = P(A)\): \(B\) carries no information about \(A\). They are conditionally independent given \(C\) when \(P(A\cap B\mid C) = P(A\mid C)P(B\mid C)\). Neither condition implies the other.

Independent, but not conditionally independent. Flip two fair coins, \(X\) and \(Y\), and let \(Z = X \oplus Y\) be their parity. \(X\) and \(Y\) are independent. Condition on \(Z=1\): now \(X=1\) forces \(Y=0\), so given \(Z\) they are perfectly dependent. Conditioning on a common effect creates dependence between its causes, the mechanism behind Berkson’s paradox and behind collider bias in observational data.

Conditionally independent, but not independent. Pick one of two coins at random, \(Z\): coin A lands heads with probability 0.9, coin B with probability 0.1. Flip the chosen coin twice, giving \(X\) and \(Y\). Given \(Z\), the flips are independent by construction. Marginally they are not: \(P(X=1)=P(Y=1)=0.5\), but \(P(X=1,Y=1) = \tfrac12(0.81) + \tfrac12(0.01) = 0.41 \ne 0.25\). Seeing heads tells you which coin you are holding, which tells you about the second flip. This is exactly the structure of a latent-variable model, conditioning on the latent decouples the observations.

Key Insight, conditioning both creates and destroys dependence: conditioning on a common cause removes dependence between its effects; conditioning on a common effect creates dependence between its causes. Which one happens is a property of the graph, not of the numbers, which is why probabilistic graphical models draw arrows before they write any probabilities.

Where this goes next

Bayes with continuous quantities needs densities rather than point masses, see random variables. The estimation view of the same machinery, priors and posteriors as inference procedures, is in statistics basics.

Recap

  • \(P(A\mid B) = P(A\cap B)/P(B)\); the chain rule and the law of total probability are immediate consequences.
  • Posterior \(\propto\) likelihood \(\times\) prior; the evidence is just the normalising sum.
  • Sensitivity 99%, specificity 95%, prevalence 1% gives a positive predictive value of exactly 1/6, not 99%.
  • Rare conditions make false positives dominate no matter how good the test is; raising prevalence to 10% moves the same test to 68.75%.
  • Independence and conditional independence imply each other in neither direction.

References

  1. Eddy, D. M. Probabilistic reasoning in clinical medicine: problems and opportunities. In Kahneman, D., Slovic, P. & Tversky, A. (eds), Judgment Under Uncertainty: Heuristics and Biases. Cambridge University Press, 1982.
  2. Tversky, A. & Kahneman, D. Judgment under uncertainty: heuristics and biases. Science 185(4157), 1124–1131, 1974.
  3. Gigerenzer, G. & Hoffrage, U. How to improve Bayesian reasoning without instruction: frequency formats. Psychological Review 102(4), 684–704, 1995.
  4. Pearl, J. Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, 1988.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.