Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Physics for Machine Learning: Why the Same Equations Keep Coming Back

5 minute read

Published:

TL;DR: Three of the most-used tools in modern generative modelling are physics with the labels changed. Diffusion models are a discretised non-equilibrium process; energy-based models are the Boltzmann distribution with a learned energy; Hamiltonian Monte Carlo is literally Hamilton's equations, and it works because of volume preservation. Symmetry arguments from Noether's theorem explain why equivariant architectures generalise. This book covers the six pieces of physics that keep reappearing, at the depth an interview actually probes.

The pattern worth noticing

An interviewer rarely asks “what is the second law of thermodynamics”. They ask why diffusion models need many small steps, or why Hamiltonian Monte Carlo uses a leapfrog integrator, or what temperature does in a softmax. Each of those has a one-line answer if you know the physics and a confused one if you do not.

The reason is historical and not accidental. Sampling from a complicated high-dimensional distribution is the central computational problem of statistical mechanics, and it has been for a century. Machine learning inherited the problem — and, with it, the solutions.

Four lines of descent

Diffusion models come from non-equilibrium thermodynamics. The founding paper, Sohl-Dickstein et al. (2015), is explicit about it: take a data distribution, drive it towards a simple equilibrium with a slow stochastic process, and learn to reverse the drive. The mathematics of “slowly destroy structure, then undo it” is the mathematics of annealed importance sampling and of the Jarzynski equality. Entropy and thermodynamics covers that ancestry; diffusion and Brownian motion covers the stochastic process itself.

Energy-based models come from statistical mechanics. Write any positive density as

\[ p_\theta(x) = \frac{e^{-E_\theta(x)}}{Z(\theta)}, \qquad Z(\theta) = \int e^{-E_\theta(x)}\,dx \]

where \(E_\theta\) is a learned energy and $Z(\theta)$ the partition function. This is the Boltzmann distribution at unit temperature. Every difficulty in training such a model — the intractable normaliser, the need for MCMC in the gradient, the role of temperature — is a difficulty statistical physics met first.

Hamiltonian Monte Carlo comes from mechanics. HMC invents a fictitious momentum, writes down a Hamiltonian, and integrates the resulting dynamics. It works because Hamiltonian flow preserves phase-space volume and is reversible, which is exactly what makes the Metropolis–Hastings acceptance ratio collapse to an energy difference. Neal’s review (2011) is the standard reference.

Symmetry arguments come from Noether. Every continuous symmetry of the action gives a conserved quantity. The modern echo is equivariant networks: build the symmetry into the architecture and the model no longer has to learn it from data.

Map from physics topics to machine learning methodsFour arrows run from a left column of physics topics to a right column of machine learning methods. Hamiltonian mechanics maps to Hamiltonian Monte Carlo. Statistical mechanics maps to energy-based models and softmax temperature. Non-equilibrium thermodynamics and Langevin dynamics map to diffusion and score-based models. Noether symmetry maps to equivariant neural networks. Physics Machine learning Hamiltonian mechanics, Liouville Boltzmann distribution, partition fn Langevin / non-equilibrium Noether: symmetry → conservation Hamiltonian Monte Carlo Energy-based models, softmax T Diffusion & score-based models Equivariant architectures
Each arrow is a derivation, not an analogy. Notice that the two columns are not related methods — the right column is the left column applied to a learned energy or a learned score.

Temperature, in one example

The clearest single case is temperature. The Boltzmann distribution over states with energies $E_i$ at temperature $T$ is $p_i \propto e^{-E_i/T}$. A softmax over logits $z_i$ is $p_i \propto e^{z_i/T}$. These are the same expression with $E_i = -z_i$. Take logits $(2.0,\,1.0,\,0.5)$:

$T$resulting probabilities
0.50.844, 0.114, 0.042
1.00.629, 0.231, 0.140
2.00.481, 0.292, 0.227

Low temperature concentrates on the lowest-energy state; high temperature flattens towards uniform. Simulated annealing, nucleus sampling and the noise schedule of a diffusion model are all versions of the same lever.

Key Insight — why physics keeps winning: physics spent a century on one problem machine learning cannot avoid: given an unnormalised density \(e^{-E(x)}\) over a space with \(10^6\) dimensions, produce samples and estimate expectations without ever computing the normaliser. Every technique that survived — Metropolis, annealing, Langevin dynamics, Hamiltonian flow — transfers unchanged when \(E\) becomes a neural network, because none of them ever needed \(E\) to have a closed form.

What the rest of the book covers

Interview trap: saying diffusion models are "inspired by" thermodynamics understates the connection and invites a follow-up you will not enjoy. The forward process is a discretised Ornstein–Uhlenbeck process, its stationary distribution is the equilibrium Gaussian, and the reverse-time SDE is a standard result from stochastic process theory. Claim the derivation, not the metaphor.

Recap

  • Sampling from an unnormalised high-dimensional density is a physics problem first and a machine learning problem second; the algorithms transferred wholesale.
  • Softmax temperature and the Boltzmann distribution are the same formula with \(E = -z\).
  • HMC works because of volume preservation and reversibility, not because Hamiltonian dynamics is a good heuristic.
  • Diffusion models descend directly from non-equilibrium thermodynamics (Sohl-Dickstein et al., 2015), not by analogy.
  • Noether's theorem is the reason symmetry constraints buy generalisation: a symmetry of the objective is a structural fact, not a regulariser.

References

  1. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015.
  2. Neal, R. M. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2011.
  3. Song, Y., & Ermon, S. Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS 2019.
  4. LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. A Tutorial on Energy-Based Learning. Predicting Structured Data, MIT Press, 2006.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.