Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

The Forward Process: A Corruption Engineered to Be Jumped Into

6 minute read

Published:

TL;DR: The forward process is a Markov chain that shrinks the image by \(\sqrt{1-\beta_t}\) and adds variance \(\beta_t\). The shrink factor is not cosmetic — without it the variance grows without bound instead of converging to \(\mathcal{N}(\mathbf{0},\mathbf{I})\). Because Gaussians compose, the whole chain collapses to a single closed-form jump governed by \(\bar{\alpha}_t\), which is what makes training cheap. The choice of schedule then decides how the signal-to-noise ratio decays, and the standard linear schedule wastes about a third of its steps on inputs that are already indistinguishable from noise.

Two things the corruption has to do

The forward process has no free parameters to learn, so it is tempting to treat it as boilerplate. It is not. It has to satisfy two constraints simultaneously, and they pull in different directions.

First, it must converge to a fixed, samplable distribution. At sampling time we start from \(\mathbf{x}_T\) drawn from a prior, so \(q(\mathbf{x}_T)\) had better actually be that prior, regardless of which image we started from.

Second, it must be jumpable. Training draws a random timestep and needs \(\mathbf{x}_t\) immediately. Simulating 400 sequential steps per training example would make the method unusable.

Why the shrink factor is there

One step of the chain is

\[ q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = \mathcal{N}\!\left(\mathbf{x}_t;\ \sqrt{1-\beta_t}\,\mathbf{x}_{t-1},\ \beta_t \mathbf{I}\right) \]

where \(\beta_t \in (0,1)\) is the variance schedule: a fixed, increasing sequence of small positive numbers. Track the per-coordinate variance \(v_t = \operatorname{Var}[\mathbf{x}_t]\). Scaling by \(\sqrt{1-\beta_t}\) multiplies variance by \(1-\beta_t\), and the injected noise adds \(\beta_t\):

\[ v_t = (1-\beta_t)\,v_{t-1} + \beta_t \]

This recursion has a fixed point at \(v = 1\), and it is attracting: \(v_t - 1 = (1-\beta_t)(v_{t-1}-1)\), so the gap to unit variance shrinks by a factor \(1-\beta_t\) every step. Data normalised to unit variance stays there; data that is not gets pulled in.

Now delete the shrink factor, so that \(\mathbf{x}_t = \mathbf{x}_{t-1} + \sqrt{\beta_t}\,\mathbf{z}\). The recursion becomes \(v_t = v_{t-1} + \beta_t\), and the variance is just an accumulating sum. With the standard linear schedule — \(T = 1000\), \(\beta\) rising linearly from \(10^{-4}\) to \(0.02\) — that sum is \(\sum_t \beta_t = 10.05\), so unit-variance data would end at variance \(11.05\), a standard deviation of \(3.32\). The terminal distribution then depends on \(T\) and on the schedule, and no fixed prior matches it. Variance-exploding formulations do exist and work, but they have to rescale explicitly at sampling time; see the score-SDE view.

The closed-form jump

Write \(\alpha_t = 1-\beta_t\) and \(\bar{\alpha}_t = \prod_{s=1}^{t}\alpha_s\). Composing \(t\) Gaussian steps gives another Gaussian, and the means and variances telescope:

\[ q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}\!\left(\mathbf{x}_t;\ \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0,\ (1-\bar{\alpha}_t)\mathbf{I}\right) \qquad\Longleftrightarrow\qquad \mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon} \]

with \(\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\). The two coefficients square to one, so \(\bar{\alpha}_t\) is exactly the fraction of the variance still carried by signal. That gives a clean scalar summary of the whole schedule, the signal-to-noise ratio:

\[ \mathrm{SNR}(t) = \frac{\bar{\alpha}_t}{1-\bar{\alpha}_t} \]

Every design question about schedules is really a question about the shape of this curve.

What the linear schedule actually does

Values for \(T=1000\), \(\beta\) linear from \(10^{-4}\) to \(0.02\):

\(t\)\(\bar{\alpha}_t\)\(\sqrt{\bar{\alpha}_t}\)\(\sqrt{1-\bar{\alpha}_t}\)\(\mathrm{SNR}(t)\)
500.97100.9850.17033.5
1000.89700.9470.3218.71
2000.65900.8120.5841.93
4000.19510.4420.8970.243
6000.02590.1610.9870.0266
8000.001530.0390.9990.00153
10000.000040.0061.0000.00004

Read the last three rows. By \(t=600\) the signal amplitude is 16% of the noise amplitude; by \(t=800\) it is 4%. Taking \(\mathrm{SNR} = 0.01\) as a generous threshold for “there is nothing recoverable here”, the linear schedule crosses it at \(t = 675\) — so 325 of the 1000 steps operate on inputs that are effectively pure noise. Those steps still cost a network evaluation at sampling time and still consume gradient updates during training.

The cosine schedule

Nichol & Dhariwal define the schedule through \(\bar{\alpha}_t\) directly rather than through \(\beta_t\):

\[ \bar{\alpha}_t = \frac{f(t)}{f(0)},\qquad f(t) = \cos^2\!\left(\frac{t/T + s}{1+s}\cdot\frac{\pi}{2}\right) \]

with a small offset \(s = 0.008\) that keeps \(\beta_1\) from collapsing to zero. The implied \(\beta_t = 1 - \bar{\alpha}_t/\bar{\alpha}_{t-1}\) is clipped at \(0.999\) to keep the final step finite. The shape is close to linear in the middle and nearly flat at both ends, so destruction is spread evenly instead of front-loaded. On the same threshold, the cosine schedule does not cross \(\mathrm{SNR}=0.01\) until \(t = 937\): 63 wasted steps instead of 325.

Log signal-to-noise ratio against timestep for the linear and cosine schedulesTwo decreasing curves on axes running from timestep 0 to 1000 horizontally and log signal-to-noise ratio from plus 6 down to minus 10 vertically. The linear schedule (orange) falls steeply and crosses log SNR of minus 4.6 at timestep 675. The cosine schedule (teal) falls almost as a straight line and does not cross minus 4.6 until timestep 937. A dashed grey line marks log SNR equal to minus 4.6, labelled "signal-to-noise ratio 0.01 — effectively pure noise". +6 0 −6 −10 0 300 600 900 timestep t log SNR SNR = 0.01 — effectively pure noise t=675 t=937 linear cosine
Notice where each curve crosses the dashed line. The linear schedule spends its last third below any useful signal level; the cosine schedule keeps signal alive almost to the end, so nearly every timestep poses a non-trivial denoising problem.
Key Insight — the schedule is a budget allocation: \(T\) is a budget of network evaluations, and the schedule decides how that budget is spread across noise levels. A schedule that drives \(\bar{\alpha}_t\) to zero early is not "more thorough" — it has simply spent a large block of its budget mapping noise to noise, at both training and sampling time. Reading the schedule as a log-SNR curve rather than as a \(\beta\) sequence makes the misallocation obvious, and it is why later work parameterises schedules by SNR directly.

When it matters and when it does not

The gap is largest at low resolution. Nichol & Dhariwal report that the linear schedule is noticeably sub-optimal at \(32\times32\) and \(64\times64\), and that under it a substantial prefix of the reverse process can be skipped with little effect on sample quality — a direct symptom of the wasted region. At \(256\times256\) the difference narrows, because a high-resolution image carries far more information and survives longer under the same relative noise.

The other consequence surfaces in DDIM and other fast samplers. When you drop from 1000 steps to 30, you subsample the schedule, and steps drawn from a dead region contribute nothing. A schedule with a well-spread log-SNR curve is a precondition for short sampling, not an independent refinement.

One residual issue: neither schedule reaches \(\bar{\alpha}_T = 0\) exactly, so \(q(\mathbf{x}_T)\) is not quite the prior we sample from. With the linear schedule \(\sqrt{\bar{\alpha}_{1000}} \approx 0.006\), a small but non-zero leak of the mean image into the terminal distribution. Zero-terminal-SNR schedules fix this by forcing \(\bar{\alpha}_T = 0\), which matters most for models expected to produce very dark or very bright outputs.

References

  1. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015.
  2. Ho, J., Jain, A., & Abbeel, P. Denoising Diffusion Probabilistic Models. NeurIPS 2020.
  3. Nichol, A., & Dhariwal, P. Improved Denoising Diffusion Probabilistic Models. ICML 2021.
  4. Lin, S., Liu, B., Li, J., & Yang, X. Common Diffusion Noise Schedules and Sample Steps are Flawed. WACV 2024.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.