Diffusion Models: Learning to Undo Noise
Published:
The asymmetry the method exploits
Generative modelling is hard because a good image is a vanishingly small target in a space of millions of pixels. Diffusion sidesteps the difficulty with a trick: instead of learning to build an image in one leap, learn to undo a corruption you designed yourself.
Destroying structure is easy. Add a little Gaussian noise, repeat a thousand times, and any image becomes indistinguishable from static. Nothing is learned in that direction — the corruption is a fixed, known process. All the learning goes into the reverse.
The forward process
Fix a variance schedule \(\beta_1, \dots, \beta_T\) with small positive values. Each step slightly shrinks the image and adds noise:
The shrink factor \(\sqrt{1-\beta_t}\) matters: without it the variance would grow without bound. With it, the process converges to a standard Gaussian regardless of where it started.
Because Gaussians compose, you never have to simulate the chain. Writing \(\alpha_t = 1-\beta_t\) and \(\bar{\alpha}_t = \prod_{s \le t}\alpha_s\), any step is reachable in closed form:
This is what makes training cheap. To get a batch at timestep \(t\), sample \(t\) at random, sample one noise vector, and apply the line above — no thousand-step simulation required.
Why the reverse is tractable
Reversing an arbitrary corruption would be hopeless. The saving fact is that when \(\beta_t\) is small, the true reverse conditional \(q(\mathbf{x}_{t-1}\mid\mathbf{x}_t)\) is itself approximately Gaussian. So the model only has to predict its mean:
This is the whole reason the step size has to be small and \(T\) large. Take steps that are too big and the reverse conditional stops being Gaussian, and a network that predicts a single mean can no longer represent it.
The objective collapses to noise prediction
Ho et al. (2020) showed that the variational bound on the likelihood, after reparameterisation and dropping weighting terms, reduces to something strikingly plain — predict the noise that was added:
It is a mean-squared error. The network sees a noisy image and a timestep, and guesses which noise vector produced it. That is the entire training loop.
A concrete pass
Take a single \(64\times64\) image and \(T = 1000\) with a linear \(\beta\) schedule from \(10^{-4}\) to \(0.02\).
| Step | What happens |
|---|---|
| Sample \(t = 400\) | \(\bar{\alpha}_{400} \approx 0.195\) |
| Build the input | \(\mathbf{x}_{400} = 0.44\,\mathbf{x}_0 + 0.90\,\boldsymbol{\epsilon}\) — mostly noise, coarse shapes survive |
| Network predicts | \(\boldsymbol{\epsilon}_\theta(\mathbf{x}_{400}, 400)\) |
| Loss | squared error against the \(\boldsymbol{\epsilon}\) actually drawn |
At \(t = 50\), \(\bar{\alpha}\approx 0.97\) and the image is nearly clean, so the network is learning fine texture. At \(t = 900\) it is nearly pure noise and the network can only recover global layout. One network handles the whole range, conditioned on \(t\) — which is why the timestep embedding matters as much as the architecture.
What this buys, and what it costs
Diffusion trains stably — it is a regression problem, with none of the adversarial balance that makes GANs fragile — and it covers modes well, because the objective is likelihood-based rather than a minimax game. The cost is sampling: a thousand sequential network evaluations to produce one image.
Almost everything that followed attacks that cost or that control problem: DDIM makes sampling deterministic and short, latent diffusion moves the process into a compressed space, classifier-free guidance buys controllability, and flow matching rebuilds the whole thing around straight paths instead of stochastic ones.
References
- Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015.
- Ho, J., Jain, A., & Abbeel, P. Denoising Diffusion Probabilistic Models. NeurIPS 2020.
- Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021.