Statistics Basics: Reasoning Backwards From Data to Model
Published:
The direction of the arrow
Almost every confusion in introductory statistics dissolves once you notice which way the arrow points.
In probability you are handed a model — a coin with bias \(\theta = 0.6\), a Gaussian with mean \(\mu\) and variance \(\sigma^2\) — and asked what data it produces. The reasoning is deductive, and given enough calculus you can write the answer down.
In statistics you are handed the data and asked about the model. That reasoning is inductive in a specific, painful way: the map from parameters to data destroys information. Seven heads in ten tosses is perfectly consistent with \(\theta = 0.7\), quite consistent with \(\theta = 0.5\), and not impossible under \(\theta = 0.9\). No amount of cleverness recovers a unique answer, because there is no unique answer to recover.
What the other seven posts do
The book splits into three groups, matching the three things you actually have to do with a sample.
Describe it. Descriptive statistics covers means against medians, variance, the reason sample variance divides by \(n-1\), skewness and kurtosis, quantiles — and Anscombe’s quartet, the standing reminder that four wildly different datasets can share every summary you compute.
Estimate from it. Estimators, bias and variance defines what makes an estimator good, derives the bias–variance decomposition of mean squared error, and shows why a deliberately biased estimator can beat an unbiased one. Maximum likelihood then gives the default recipe for constructing estimators, works it through for a Gaussian, and connects it to KL divergence, to MAP estimation, and to L2 regularisation.
Quantify how sure you are. Confidence intervals builds the interval and, more importantly, states precisely what it claims. Hypothesis testing covers p-values, error types, power, and multiple comparisons. The bootstrap gets uncertainty when the algebra runs out, and explains cross-validation as the same idea. Bayesian versus frequentist closes by showing what the two schools each actually claim, and when the numbers agree anyway.
Probability itself — random variables, expectation, the distributions, the limit theorems — is assumed. If any of that is rusty, start from the probability basics overview and come back.
Why estimation and uncertainty matter most in ML
Machine learning is estimation. A trained network is a point estimate of a parameter vector chosen to fit a sample, and every pathology of estimation appears in it. Overfitting is estimator variance. Regularisation is deliberate bias traded for a variance reduction. A validation score is a statistic with its own sampling distribution, which is why a 0.3-point gap on a 2,000-example test set means almost nothing.
Uncertainty is the part that gets dropped, and dropping it is expensive: leaderboards that do not replicate, A/B tests that ship noise, and reported metrics whose third digit is not even stable.
How to use this book
Read it in order; each post assumes the previous ones. If you are revising against the clock, the load-bearing derivations are Bessel’s correction, the bias–variance decomposition, and the Gaussian MLE — those three are asked about far more than anything else, and all three are short enough to reproduce on a whiteboard.
Recap
- Probability maps model to data and is deductive; statistics maps data to model and is inductive and one-to-many.
- Because the inverse is ambiguous, every honest answer has two parts: an estimate and a quantified uncertainty.
- Every frequentist method reasons about the sampling distribution — what the estimator would do across repeated samples from a hypothesised model.
- ML is estimation: overfitting is variance, regularisation is bias bought on purpose, and validation scores are noisy statistics.
- The classic interview failures are interpretive: confidence intervals and p-values both make statements about the procedure, not about the parameter.
References
- Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer, 2004.
- Efron, B., & Hastie, T. Computer Age Statistical Inference. Cambridge University Press, 2016.
- Hastie, T., Tibshirani, R., & Friedman, J. The Elements of Statistical Learning, 2nd ed. Springer, 2009.
- Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31, 337–350, 2016.