Probability for ML Interviews: The Eight Ideas Worth Re-deriving

Why the language matters

You can train a classifier without ever writing down a likelihood. You cannot explain one. The moment an interviewer asks why the softmax cross-entropy loss is the right thing to minimise, or what a model’s 0.83 output actually means, the answer is probabilistic and there is no way around it.

Three threads run through most of contemporary machine learning, and all three are probability.

Likelihoods. Fitting a model by maximum likelihood means choosing parameters $\theta$ to make the observed data as probable as possible under \(p_\theta\). Take the negative logarithm and you have a loss:

Here \(p_\theta(y\mid x)\) is the model’s predicted distribution over labels. Instantiate it as a categorical distribution and this is softmax cross-entropy; instantiate it as a Gaussian with fixed variance and it is mean squared error. The two losses everyone uses are the same principle with two different assumed noise models.

Generative models. A generative model is a claim about $p(x)$, and every architecture is a different way of making that claim tractable. Autoregressive models factor it with the chain rule, \(p(x) = \prod_t p(x_t \mid x_{1:t-1})\). Diffusion models define a Markov chain of conditionals and learn to reverse it. Variational autoencoders bound the intractable $\log p(x)$ using a KL divergence. None of these are readable without conditioning and divergences.

Uncertainty. A deployed model that says “cat, 0.99” when it is looking at noise is not making a modelling error so much as a probabilistic one. Distinguishing what the model does not know because the data are noisy from what it does not know because it has never seen this input requires the decomposition in the law of total variance.

The map

The other seven chapters go in dependency order. Nothing later assumes anything you have not already read.

Sample spaces and the axioms sets up the object itself: outcomes, events, and the three Kolmogorov axioms. It explains why countable additivity is the interesting axiom, why not every subset can be assigned a probability, and it works inclusion–exclusion through a counting example.

Conditional probability and Bayes is the chapter that earns its keep in interviews. The chain rule, the law of total probability, Bayes’ theorem — then a medical test with 99% sensitivity where a positive result still means only a one-in-six chance of disease. It also separates independence from conditional independence, which are unrelated properties despite the shared word.

Random variables moves from events to numbers: PMFs, PDFs, CDFs, and the fact that a density value is not a probability and can exceed one. Then change of variables with the Jacobian — the identity underneath normalising flows — and inverse-transform sampling.

Expectation and variance covers the moments. Linearity of expectation holds without independence, which is the single most useful fact in combinatorial probability. Variance does not, which is why the covariance term exists. Ends with a two-line counterexample to “uncorrelated implies independent”.

Common distributions is a reference table — ten distributions with parameters, means and variances — plus real treatment of the multivariate Gaussian, whose covariance matrix determines the geometry of its level sets.

Limit theorems covers the law of large numbers, the central limit theorem and what it does not claim, the three modes of convergence, and the concentration inequalities that give you finite-sample guarantees where the CLT only gives asymptotics.

Information theory closes the loop: entropy, cross-entropy, KL divergence and its asymmetry, mutual information, and why forward KL covers modes while reverse KL seeks them.

How to use this book

Read a chapter, then close it and re-derive the boxed results. The traps in this material are not hard theorems — they are places where an intuitive answer is confidently wrong: base rates get ignored, zero correlation gets read as independence, a density gets read as a probability, and the CLT gets applied to $n = 8$. Each chapter flags its own in a warning box.

For the estimation side of the same story — estimators, bias and variance, confidence intervals, hypothesis testing — see the companion book at statistics basics.

References

1. Murphy, K. P. Probabilistic Machine Learning: An Introduction. MIT Press, 2022.
2. Bishop, C. M. Pattern Recognition and Machine Learning. Springer, 2006.
3. Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer, 2004.
4. MacKay, D. J. C. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003.