Sample Spaces and the Kolmogorov Axioms: What a Probability Actually Is
Published:
The three objects
| Start with the sample space $\Omega$: the set of every outcome the experiment could produce, with exactly one occurring. Two dice give \(\Omega = \{1,\dots,6\}^2\), so $ | \Omega | = 36$. A real-valued measurement gives $\Omega = \mathbb{R}$. |
An event is a subset of $\Omega$ — “the sum is 7” is the six-element set \(\{(1,6),(2,5),\dots,(6,1)\}\). Events are sets, which is why probability inherits the algebra of sets: “and” is intersection, “or” is union, “not” is complement. The collection of admissible events is $\mathcal{F}$, closed under complement and countable union.
Probability is a function $P:\mathcal{F}\to[0,1]$ obeying three axioms:
Axiom (iii) is countable additivity: the probability of a union of pairwise disjoint events is the sum of their probabilities, for infinitely many events, not just finitely many. Nothing here says what a probability means; frequentist and Bayesian readings are interpretations layered on top of the same axioms.
Why countable, and why not all subsets
Finite additivity alone is too weak. Ask for the probability that a fair coin flipped repeatedly ever shows heads. Let $A_i$ be “first head on flip $i$”, so $P(A_i)=2^{-i}$, and the events are disjoint. Only countable additivity licenses
The same axiom gives continuity: if $A_1 \subseteq A_2 \subseteq \cdots$ then \(P(\bigcup_i A_i) = \lim_{n} P(A_n)\). Every limiting argument in probability — the law of large numbers, almost-sure convergence, the behaviour of a Markov chain as $t\to\infty$ — runs through that exchange of limit and probability, and it is countable additivity that permits it.
The other restriction is subtler. You might hope $\mathcal{F} = 2^{\Omega}$, every subset. For a countable $\Omega$ it can be. For $\Omega = [0,1]$ with the uniform distribution it cannot: Vitali showed in 1905 that one can construct a set whose translates tile $[0,1]$ in a way incompatible with any translation-invariant, countably additive assignment of length. Assigning it any value at all — zero, positive, anything — contradicts the axioms. The fix is to shrink $\mathcal{F}$ to the Borel sets: the smallest $\sigma$-algebra containing the intervals. Every set you will meet in practice is Borel, which is why the restriction is invisible in applications. A random variable is then just a function $X:\Omega\to\mathbb{R}$ for which \(\{X \le t\}\) lies in $\mathcal{F}$ for all $t$ — “measurable” means “you are allowed to ask for its probability”.
Consequences you use constantly
From the axioms alone: $P(A^c) = 1 - P(A)$; if $A \subseteq B$ then $P(A) \le P(B)$; and for two events
The subtraction corrects the double-count of the overlap. Extended to $n$ events, this is the inclusion–exclusion principle: add all singles, subtract all pairwise intersections, add back all triples, and so on with alternating sign:
Truncating after the first term gives the union bound $P(\bigcup_i A_i) \le \sum_i P(A_i)$, which needs no independence and is the workhorse of generalisation bounds and multiple-testing corrections.
A worked count
Draw an integer uniformly from \(\{1,\dots,100\}\). What is the probability it is divisible by 2, 3 or 5?
| Let $A_d$ be “divisible by $d$”, so $ | A_d | = \lfloor 100/d \rfloor$. Intersections are divisibility by the products, since 2, 3 and 5 are coprime. |
| Term | Set | Count |
|---|---|---|
| singles | $A_2, A_3, A_5$ | $50 + 33 + 20 = 103$ |
| pairs | \(A_6, A_{10}, A_{15}\) | $-(16 + 10 + 6) = -32$ |
| triple | \(A_{30}\) | $+3$ |
| total | 74 |
So $P = 74/100 = 0.74$. Naively adding the singles would have given $1.03$ — impossible, and a clean diagnostic that overlaps were double-counted. The union bound here says only $P \le 1.03$, which is true and useless: bounds get loose exactly when events overlap heavily.
Where this goes next
Once $P$ exists, conditioning restricts it to a sub-event and renormalises — that is conditional probability and Bayes. Pushing $P$ forward through a measurable function gives a distribution on $\mathbb{R}$, which is what random variables are about.
Recap
- A probability space is \((\Omega,\mathcal{F},P)\); the axioms are non-negativity, \(P(\Omega)=1\), and countable additivity.
- Countable additivity gives continuity of \(P\), which every limit theorem depends on.
- \(\mathcal{F}\) cannot be all subsets of \([0,1]\) — non-measurable sets exist — so events are restricted to a \(\sigma\)-algebra.
- Inclusion–exclusion alternates signs over intersections; truncating it after one term gives the union bound.
- Probability zero means "almost never", not "impossible".
References
- Kolmogorov, A. N. Grundbegriffe der Wahrscheinlichkeitsrechnung. Springer, 1933. (English: Foundations of the Theory of Probability, Chelsea, 1956.)
- Billingsley, P. Probability and Measure, 3rd ed. Wiley, 1995.
- Durrett, R. Probability: Theory and Examples, 5th ed. Cambridge University Press, 2019.
- Feller, W. An Introduction to Probability Theory and Its Applications, Vol. 1, 3rd ed. Wiley, 1968.