Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Machine Learning Before Transformers: A Working Foundation

4 minute read

Published:

What this book is for: the later books — Transformers, Graph Neural Networks, Sheaf models, Diffusion — all assume a working knowledge of losses, gradients, regularisation, and the pre-Transformer architectures. This book supplies exactly that, and no more. It is not a survey; it is the shortest path to being able to read the rest.

One thread runs through all of it

It is tempting to read the history of machine learning as a list of unrelated tricks. It reads better as a single question asked repeatedly:

How much structure do we build into the model, and how much do we let the data decide?

Every method in this book is an answer to that. Linear regression builds in a great deal — it asserts the relationship is a straight line — and in exchange needs very little data and yields coefficients you can read. A Transformer builds in almost nothing about which positions should interact, and in exchange needs a great deal of data. Everything else sits between those poles.

Model families ordered by how much structure they assumeSix families from linear models to Transformers, each labelled with the prior it builds in, ordered left to right from most assumed structure to most learned from data. Each family is a different answer to “how much do we assume?” Linear models Trees · kernels Unsupervised Neural nets CNNs · RNNs Transformers assumes the relationship is linear assumes similar inputs give similar outputs assumes there is structure without any labels assumes the features themselves can be learned assumes locality (CNN) or sequential order (RNN) assumes almost nothing — learns what interacts more structure assumed · less data needed · easier to interpret more learned from data · more data needed · harder to interpret Nothing here is obsolete: the left of this line is still the right answer for small, tabular, or interpretability-critical problems.
The organising axis of this book. Moving right buys flexibility and costs data, compute, and interpretability — a trade, not an upgrade.
The direction of that arrow is not a ranking. On a few hundred rows of tabular data, gradient-boosted trees routinely beat anything deeper, and a logistic regression you can explain to a regulator may be worth more than two points of accuracy you cannot. “Newer” is a statement about assumptions, not about quality.

What you need before the later books

Each book on this site leans on a specific part of this one:

  • Transformers assumes softmax, cross-entropy, gradient descent, residual connections, and — most of all — that you know what a convolution’s fixed receptive field costs, because attention is the answer to that cost.
  • Graph Neural Networks assume message passing is a generalisation of convolution, which only lands if convolution is solid first.
  • Sheaf Neural Networks assume the graph Laplacian, eigen-decomposition, and the idea of a diffusion operator.
  • Diffusion models assume maximum likelihood, the Gaussian, and the ELBO.

The chapters

Reading order: the chapters are arranged by dependency, not by difficulty. Gradient descent comes first because everything after it is trained that way; linear regression comes next because it is the one model where you can see the whole solution in closed form and check it by hand.

Learning foundations — how a model is trained at all: the gradient descent update, why the learning rate can break everything, and backpropagation as reverse-mode differentiation.

Supervised learning — linear regression as a projection, logistic regression as linearity in the log-odds, and the classical non-linear workhorses.

Unsupervised learning — clustering, where the real lesson is that each algorithm encodes a different definition of what a cluster is, and dimensionality reduction.

Neural networks — the multilayer perceptron, then convolutions as a fully-connected layer with locality and weight sharing imposed, and recurrent models for sequences.

Activation functions — the three chapters already published, covering why non-linearity is required at all, the modern smooth-gating family, and output activations as a contract with your loss.

Status. This book is being written now, and chapters appear as they are finished. The cards below this overview show what is live. If a topic you want is missing, it is likely in the next batch rather than deliberately omitted.

How to read it

If you are here to unblock a later book, read only what that book needs — the list above tells you which chapters those are. If you are building the foundation properly, read in order; each chapter assumes the ones before it and nothing else.

Every chapter follows the same shape: the intuition first, then the mathematics, then a worked example small enough to check by hand, then an honest section on what the method does not do. That last part is the one worth reading twice.

✅ Key Takeaways

  • This book exists to make the other books readable, not to survey the field.
  • The through-line is how much structure a model assumes versus how much it learns — that single axis orders everything from least squares to attention.
  • Moving along that axis is a trade against data, compute, and interpretability. It is not a ranking, and the classical end is still the correct choice for a great many real problems.
  • Chapters are ordered by dependency. Gradient descent first, because everything afterwards is trained with it.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed — most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually — but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first — which is faster, shorter, and free of race conditions.