Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

What Mathematics an ML Engineer Actually Needs

6 minute read

Published:

TL;DR: A trained network is a composition of linear maps and pointwise nonlinearities, fitted by following gradients. That single sentence determines the syllabus. You need linear algebra to understand what the linear maps do, and multivariable calculus to understand how the gradients are computed. Norms and convexity round it out because they tell you when the fitting converges. Measure theory, topology and real analysis are not what you will be asked about — and this book says so rather than padding itself with them.

The syllabus is determined by one sentence

Write down what a feed-forward network is. Each layer applies a matrix, adds a bias, and passes the result through a nonlinearity applied elementwise. Training adjusts the matrices by descending a scalar loss.

Everything mathematical you need follows from unpacking that:

  • The matrices. What does $W$ do to a vector? Which directions does it stretch, which does it destroy, what information does it throw away? That is linear algebra, and it is the single largest slice of the syllabus.
  • The descending. How does a change in one weight, buried eleven layers deep, affect a scalar at the end? That is the chain rule, applied at scale. Backpropagation is nothing else.
  • The convergence. How fast does this get anywhere, and what shape of loss surface makes it slow? That is norms, curvature and convexity.

The nonlinearity is the only piece that is not mathematically interesting, which is why it gets one line and moves on.

The honest note about interviews

Interview mathematics is overwhelmingly linear algebra and calculus. Rank, eigenvectors, SVD, gradients, Jacobians, and a sanity check on convexity account for the large majority of questions actually asked. Probability and statistics form their own body of material and are treated separately from this book.

What almost never appears: epsilon-delta arguments, Lebesgue integration, compactness, Banach spaces. If a role genuinely needs those — theory positions, some optimisation research — the interview will say so in advance. Revising analysis instead of practising a Jacobian derivation is a bad trade.

What does appear, and catches people out, is not hard material. It is material people once knew and then stopped using: the difference between eigenvalues and singular values, why the gradient of $x^\top A x$ is $(A + A^\top)x$ and not $Ax$, why $L^1$ produces sparsity and $L^2$ does not.

The three traps this book keeps returning to: (1) treating eigenvalues and singular values as the same object — they agree only for symmetric positive semi-definite matrices, and the general relationship is that the singular values of \(A\) are the square roots of the eigenvalues of \(A^\top A\); (2) getting matrix-calculus layout wrong, which produces an answer that is right up to a transpose and therefore wrong; (3) claiming a stationary point is a minimum without checking curvature.

The map

The seven remaining posts, in the order they build on one another.

Linear algebra

  1. Vectors and matrices — span, basis, rank; matrix multiplication read as composition of maps and as a change of basis; column space against null space, and rank–nullity as the conservation law that connects them.
  2. Eigenvalues, the spectral theorem and SVD — invariant directions, when diagonalisation fails, why the SVD always exists, and low-rank approximation with a worked $2\times2$ example carried through exactly.

Calculus

  1. Derivatives and gradients — the derivative as the best linear approximation, the gradient as a covector, and why reverse-mode differentiation is cheap precisely when there is one scalar output.
  2. Jacobians and Hessians — change of variables and the log-determinant term in normalising flows; curvature, second-order optimality, and why Newton’s method loses at scale.
  3. Matrix calculus — the dozen identities worth memorising, one derivation done slowly, and the layout convention that causes most sign errors.

Analysis and optimisation

  1. Norms and distances — $L^1$, $L^2$, $L^\infty$, operator and Frobenius norms; the geometry behind $L^1$ sparsity; cosine similarity against Euclidean distance.
  2. Convexity and optimisation — what convexity guarantees, how the condition number sets the convergence rate, KKT conditions, and the honest position on non-convex deep learning.
Key Insight — the two views that unify the book: almost every result here is one of two statements in disguise. Either "this matrix is a map, so ask what it does to space" — which gives you rank, eigenvectors, SVD, operator norms and conditioning — or "this function is locally linear, so approximate it" — which gives you gradients, Jacobians, Hessians, Newton steps and convergence rates. If you can place a question in one of those two frames, you can usually reconstruct the answer instead of recalling it.

How to use this for revision

Read each post once for the argument, then close it and try to reproduce the one worked example from memory. The examples are chosen to be small enough to do by hand and specific enough that getting them right means you understood the mechanism, not the vocabulary.

Recap

  • The syllabus falls out of one fact: a network is composed linear maps trained by gradients. Linear algebra explains the maps, calculus explains the gradients.
  • Interview mathematics is dominated by linear algebra and calculus. Real analysis is rarely asked and is a poor use of revision time.
  • Two framings recover most answers: what does this matrix do to space, and what is the local linear approximation.
  • The recurring traps are eigenvalues versus singular values, matrix-calculus layout, and asserting optimality from a vanishing gradient alone.

References

  1. Strang, G. Introduction to Linear Algebra, 6th ed. Wellesley-Cambridge Press, 2023. Companion lectures: MIT 18.06.
  2. Deisenroth, M. P., Faisal, A. A., & Ong, C. S. Mathematics for Machine Learning. Cambridge University Press, 2020.
  3. Boyd, S., & Vandenberghe, L. Convex Optimization. Cambridge University Press, 2004.
  4. Golub, G. H., & Van Loan, C. F. Matrix Computations, 4th ed. Johns Hopkins University Press, 2013.
  5. Petersen, K. B., & Pedersen, M. S. The Matrix Cookbook, version 20121115, 2012.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.