Machine Learning Before Transformers: A Working Foundation

One thread runs through all of it

It is tempting to read the history of machine learning as a list of unrelated tricks. It reads better as a single question asked repeatedly:

How much structure do we build into the model, and how much do we let the data decide?

Every method in this book is an answer to that. Linear regression builds in a great deal, it asserts the relationship is a straight line, and in exchange needs very little data and yields coefficients you can read. A Transformer builds in almost nothing about which positions should interact, and in exchange needs a great deal of data. Everything else sits between those poles.

What you need before the later books

Each book on this site leans on a specific part of this one:

• Transformers assumes softmax, cross-entropy, gradient descent, residual connections, and, most of all, that you know what a convolution’s fixed receptive field costs, because attention is the answer to that cost.
• Graph Neural Networks assume message passing is a generalisation of convolution, which only lands if convolution is solid first.
• Sheaf Neural Networks assume the graph Laplacian, eigen-decomposition, and the idea of a diffusion operator.
• Diffusion models assume maximum likelihood, the Gaussian, and the ELBO.

The complete reading map

The book now contains 42 chapters across seven sections, plus this overview. Survey chapters introduce the families; dedicated chapters derive individual methods and explain their limits.

Learning Foundations & Experimental Practice

• Bias, Variance, and Regularisation — A model can fail for two opposite reasons: it is too rigid to represent the truth, or so flexible that it chases the noise. Squared error splits cleanly into exactly those two terms plus a floor you cannot beat.
• Gradient Descent and Backpropagation — Training is one loop: measure the loss, ask backpropagation which way is downhill, take a small step. This chapter derives exactly how small that step has to be, why the answer is 2/a for a quadratic, and why reverse-mode differentiation gets you every gradient for roughly the price of one forward pass.
• ML Preprocessing — Work through training-only scaling, distinguish missingness strategies, and build preprocessing decisions around the distance, loss and deployment information available.
• Cross-Validation — Design random, grouped and temporal splits, compute an out-of-fold error correctly, and separate model selection from evaluation of the selection procedure.

Linear, Local & Probabilistic Prediction

• Linear Regression — Fitting a line by least squares is not an optimisation trick, it is the orthogonal projection of the observation vector onto the column space of the design matrix, and once you see that, the normal equations, the failure modes, and the reason we solve by QR instead of inverting all follow from one picture.
• Ridge Regression — Derive the regularised least-squares solution, inspect shrinkage in the SVD basis, and connect scaling, intercept handling and validation to the meaning of the penalty.
• Lasso Regression — Derive the coordinate update from the L1 subgradient, calculate exact zeros, and explain when sparse selection becomes unstable or elastic net is preferable.
• Logistic Regression — Logistic regression is not a squashed linear regression, it is a straight line drawn in log-odds space, which is why one coefficient means one multiplication of the odds, and why perfectly separable data drives that coefficient to infinity.
• K-Nearest Neighbours — Compute classification and regression predictions, analyse the neighbourhood-size tradeoff, and understand why representation and search strategy dominate this simple estimator.
• Naive Bayes — Derive the class score, calculate a smoothed text example, and distinguish Gaussian, multinomial and Bernoulli likelihoods and their failure modes.
• Support Vector Machines — Among all the hyperplanes that separate two classes, one sits furthest from both. Finding it turns out to depend on the data only through inner products, and that single fact is what lets you work in a space you never build.

Decision Trees & Ensembles

• Trees, Forests, and Boosting — A decision tree chops feature space into axis-aligned boxes and predicts one number per box, which explains why it needs no feature scaling, why it approximates a diagonal boundary as a staircase, and why it cannot extrapolate a single step beyond the training range.
• Decision Trees — Calculate Gini and regression split improvements, understand greedy axis-aligned partitions, and connect leaf size and pruning to prediction stability.
• Random Forests — Derive the variance of correlated averages, calculate the out-of-bag fraction, and explain why random feature subsets and diagnostic choices matter.
• AdaBoost — Derive binary AdaBoost’s learner weight, compute a complete reweighting step, and explain how exponential loss connects the algorithm to margins and label-noise sensitivity.
• Gradient Boosting — Derive residual fitting from the loss gradient, work through a regression update, and extend the mechanism to logistic and second-order tree boosting.

Clustering: One Method at a Time

• Clustering — k-means says a cluster is a ball around a centroid, DBSCAN says it is a connected dense region, and a Gaussian mixture says it is a bump in a density, pick the algorithm and you have already picked the answer.
• K-means — Derive the assignment and centroid updates, run a complete example, understand k-means++ seeding, and separate a good objective value from a useful clustering.
• Mini-Batch K-means — Derive the running-mean update, distinguish mini-batch optimisation from concept-drift tracking, and measure the speed–quality tradeoff on a fixed evaluation sample.
• K-medoids — Build a clustering around actual observations, calculate medoid costs, and understand the robustness and computational tradeoffs of partitioning around medoids.
• Agglomerative Clustering — Compute competing linkage decisions, derive Ward’s merge cost, and read a dendrogram without confusing a greedy hierarchy with a unique natural taxonomy.
• DBSCAN — Trace density-connected components by hand, distinguish core and border observations, and understand why one neighbourhood scale can fail on varying densities.
• OPTICS — Compute core and reachability distances, interpret valleys in the ordering, and distinguish the OPTICS representation from the rule used to extract a partition.
• HDBSCAN — Build the density hierarchy from mutual reachability, explain cluster persistence and minimum-size pruning, and interpret membership strengths without calling them class probabilities.
• Gaussian Mixture Models — Derive expectation–maximisation updates, calculate a soft assignment, and diagnose likelihood singularities, covariance assumptions and model-selection errors.
• Spectral Clustering — Derive the graph-smoothness objective, connect eigenvectors to disconnected components, and explain why affinity construction often matters more than the final k-means step.
• Mean Shift — Derive the weighted-mean movement behind mode seeking, follow a local update, and explain why bandwidth and mode merging determine the final clustering.
• BIRCH — Derive the additive clustering-feature summary, calculate a subcluster radius, and understand what a CF tree preserves and loses before global clustering.
• Fuzzy C-means — Derive the membership and centre updates, calculate a two-centre assignment, and distinguish fuzzy membership from a probabilistic mixture posterior.
• Evaluating Clustering — Calculate silhouette and pair agreement, diagnose misleading cluster-count criteria, and design evaluations that match the purpose of an unsupervised analysis.

PCA, Factorisation & Neighbourhood Embeddings

• PCA — PCA can be derived by asking for the directions of greatest spread, or by asking for the subspace that loses the least when you project onto it. The two questions look unrelated and have the same answer, which is the most useful thing to understand about it.
• Kernel PCA — Derive kernel centring and feature-space coordinates, project new observations correctly, and distinguish nonlinear representation from reconstruction in the original space.
• Independent Component Analysis — Explain the linear mixing model, why decorrelation is insufficient, and how non-Gaussian contrasts identify sources only under explicit assumptions and ambiguities.
• Nonnegative Matrix Factorisation — Derive nonnegative low-rank reconstruction, inspect multiplicative updates and scale ambiguity, and separate useful additive factors from guaranteed semantic parts.
• Random Projections — Derive expected squared-distance preservation, interpret finite-set concentration bounds, and compare random embeddings with PCA and neighbourhood-based methods.
• t-SNE — Derive high- and low-dimensional affinities, explain the asymmetric KL objective, and identify which distances and apparent clusters a t-SNE plot does not establish.
• UMAP — Explain local distance normalisation, fuzzy graph combination and attractive–repulsive layout, then connect neighbour count and minimum distance to practical interpretation.

Neural Networks

• Convolutions and CNNs — A dense layer from a 224-by-224 colour image to 1000 units holds 150.5 million weights; a 3-by-3, 64-filter convolution holds 1,792, and the two constraints that buy that factor of 84,000 are exactly the prior that makes it work on images.
• RNNs, LSTMs, and GRUs — A recurrent network shares weights across time exactly as a convolution shares them across space. The trouble is that gradients then travel through a product of Jacobians, and a product of a hundred numbers slightly below one is zero.

Activation Functions

• Activation Functions in Neural Networks — Activation functions are the reason neural networks can model curved decision boundaries instead of collapsing into one giant linear map. This chapter builds the intuition first, then walks through the classical functions that shaped deep learning.
• Modern Activation Functions — Once ReLU became the default, researchers started asking a better question: can we keep the easy optimization while making the activation smoother, softer, and more expressive? This chapter covers the modern answers.
• Output and Gated Activations — The last activation in your network is not a modelling preference, it is a contract with your loss function. Break it and training stops meaning anything. Here is the contract, and what changes when the activation itself becomes learned.

How to read it

If you are here to unblock a later book, read only what that book needs, the list above tells you which chapters those are. For a broader foundation, begin with learning, preprocessing and validation, then follow the method family that matches the problem. Use the dedicated clustering evaluation chapter to compare partitions, and the PCA chapter before its nonlinear alternatives.

Every chapter follows the same shape: the intuition first, then the mathematics, then a worked example small enough to check by hand, then an honest section on what the method does not do. That last part is the one worth reading twice.