Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

DDIM: Same Marginals, Fewer Steps, and a Latent Space Worth Having

6 minute read

Published:

TL;DR: The DDPM loss depends on the forward process only through the marginals \(q(\mathbf{x}_t\mid\mathbf{x}_0)\). A one-parameter family of non-Markovian forward processes shares those marginals exactly, so a trained DDPM is simultaneously the correct model for all of them — no retraining. The parameter \(\eta\) sets how much noise each reverse step injects: \(\eta=1\) recovers DDPM, \(\eta=0\) is a deterministic map. Determinism is what makes 20–50 steps viable and what turns \(\mathbf{x}_T\) into a usable latent code.

The assumption nobody needed

Look back at what training actually consumes. A batch is built as \(\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}\), the network predicts \(\boldsymbol{\epsilon}\), and that is all. The intermediate variables \(\mathbf{x}_1,\dots,\mathbf{x}_{t-1}\) never appear. The Markov chain was scaffolding used to derive the marginals; the objective only ever sees the marginals themselves.

So: is there another forward process with the same \(q(\mathbf{x}_t\mid\mathbf{x}_0)\) but a different joint? There is a continuum of them. Song et al. define, for any non-negative \(\sigma_t\),

\[ q_\sigma(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0) = \mathcal{N}\!\left( \sqrt{\bar{\alpha}_{t-1}}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_{t-1}-\sigma_t^2}\cdot\frac{\mathbf{x}_t - \sqrt{\bar{\alpha}_t}\mathbf{x}_0}{\sqrt{1-\bar{\alpha}_t}}, \ \sigma_t^2\mathbf{I}\right) \]

Conditioning on \(\mathbf{x}_0\) throughout is what breaks the Markov property: \(\mathbf{x}_{t-1}\) depends on both its successor and the original image. The construction is arranged so the marginals survive. Substituting \(\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0+\sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}\), the fraction collapses to \(\boldsymbol{\epsilon}\) and

\[ \mathbf{x}_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_{t-1}-\sigma_t^2}\,\boldsymbol{\epsilon} + \sigma_t\mathbf{z} \]

The two independent noise terms have variances summing to \((1-\bar{\alpha}_{t-1}-\sigma_t^2) + \sigma_t^2 = 1-\bar{\alpha}_{t-1}\), exactly as required, for every \(\sigma_t\). The mean is \(\sqrt{\bar{\alpha}_{t-1}}\mathbf{x}_0\) regardless. The whole family is consistent with one trained network.

The dial

Replace \(\mathbf{x}_0\) with the model’s estimate \(\hat{\mathbf{x}}_0 = (\mathbf{x}_t - \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}_\theta)/\sqrt{\bar{\alpha}_t}\) and you have a sampler:

\[ \mathbf{x}_{t-1} = \underbrace{\sqrt{\bar{\alpha}_{t-1}}\,\hat{\mathbf{x}}_0}_{\text{predicted signal}} + \underbrace{\sqrt{1-\bar{\alpha}_{t-1}-\sigma_t^2}\;\boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t)}_{\text{direction pointing to }\mathbf{x}_t} + \underbrace{\sigma_t\mathbf{z}}_{\text{fresh noise}}, \qquad \sigma_t = \eta\sqrt{\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t}}\sqrt{1-\frac{\bar{\alpha}_t}{\bar{\alpha}_{t-1}}} \]

At \(\eta=1\) the coefficient is \(\sqrt{\tilde{\beta}_t}\), the DDPM posterior standard deviation, and the update reduces to ancestral sampling. At \(\eta=0\) the noise term vanishes and the step is a deterministic function of \(\mathbf{x}_t\).

Nothing here is retrained. The same weights, the same \(\bar{\alpha}\) table, a different read-out.

Why determinism buys short schedules

The subsequence \(\tau_1<\dots<\tau_S\) need not be all of \(1..T\); take every twentieth index and you have a 50-step sampler. The formula above works for any pair of adjacent indices. But the two settings of \(\eta\) degrade very differently when you stride, because \(\sigma_t\) grows with the gap. On the linear schedule with a stride of 20:

step \(t\to t_{\text{prev}}\)\(\sigma_t\) at \(\eta=1\)\(\sigma_t\) at \(\eta=0\)
\(1000\to980\)0.5740
\(600\to580\)0.4590
\(300\to280\)0.3190
\(100\to80\)0.1570

Read the top row. At \(t=1000\to980\) the total noise amplitude is \(\sqrt{1-\bar{\alpha}_{980}}\approx 1\), and \(\eta=1\) makes \(0.574\) of that freshly injected — about a third of the variance in \(\mathbf{x}_{t-1}\) is randomness added after the network had just worked to remove it. Over a 1000-step schedule that is fine, because a thousand subsequent steps remove it again. Over 50 there are not enough, and samples come out visibly noisy. At \(\eta=0\) there is nothing to remove: every step is pure progress. Song et al. report sample quality comparable to full DDPM using 10× to 50× fewer steps.

There is a deeper reason. With \(\eta=0\), dividing through by \(\sqrt{\bar{\alpha}_{t-1}}\) and writing \(\bar{\mathbf{x}} = \mathbf{x}/\sqrt{\bar{\alpha}}\), \(\varsigma = \sqrt{1-\bar{\alpha}}/\sqrt{\bar{\alpha}}\) turns the update into \(\bar{\mathbf{x}}_{t-1} - \bar{\mathbf{x}}_t = (\varsigma_{t-1}-\varsigma_t)\,\boldsymbol{\epsilon}_\theta\) — a plain Euler step on the ODE \(d\bar{\mathbf{x}} = \boldsymbol{\epsilon}_\theta\,d\varsigma\). DDIM is a first-order solver for the probability-flow ODE, and step count is a discretisation error, not a source of accumulated noise. Stochastic samplers have no such reserve to draw on.

Stochastic versus deterministic reverse trajectories from a single latentTwo stacked panels, each starting from the same point labelled x-T on the left. In the upper panel, marked eta equals 1, three jagged paths leave the starting point and end at three different images on the right, showing that one latent maps to many outputs. In the lower panel, marked eta equals 0, a single smooth path leaves the starting point and ends at exactly one image, showing that the map from latent to image is a function and can be inverted. η = 1 (DDPM) η = 0 (DDIM) x_T x_T many one reverse time →
The same latent, run twice. Under \(\eta=1\) the fresh noise at each step means \(\mathbf{x}_T\) determines only a distribution over images; under \(\eta=0\) it determines exactly one, which is what makes the latent space addressable at all.

A latent space you can navigate

Once sampling is a deterministic map \(\Phi:\mathbf{x}_T\mapsto\mathbf{x}_0\), the noise vector stops being a scratch variable and becomes an encoding. Running the same Euler steps forwards in \(\varsigma\) inverts it, mapping a real image to the latent that reproduces it — DDIM inversion, the basis of most training-free editing methods. The inversion is approximate: each step linearises \(\boldsymbol{\epsilon}_\theta\) locally, and the error grows as the step count falls, which is why editing pipelines invert with more steps than they sample with.

Key Insight — interpolate on the sphere, not the segment: in \(d\) dimensions a standard Gaussian concentrates on a shell of radius \(\sqrt{d}\). For a \(3\times64\times64\) image, \(d = 12{,}288\) and \(\lVert\mathbf{x}_T\rVert \approx 110.9\). The midpoint of two independent samples has expected norm \(\sqrt{d/2}\approx 78.4\) — a 29% shrink, far outside the shell the model ever saw. Linear interpolation therefore drifts into low-density latents and yields washed-out images. Spherical interpolation, \(\mathrm{slerp}(\mathbf{x}^{(0)},\mathbf{x}^{(1)};\lambda)\), stays on the shell, and the resulting image sequences move through semantic attributes rather than cross-fading pixels.

What it does not fix

Determinism costs diversity: with \(\eta=0\) all stochasticity lives in \(\mathbf{x}_T\), and the reverse pass cannot correct an error once made. Stochastic samplers self-correct, because injected noise gives the score field further chances to pull a drifting trajectory back — which is why Karras et al. find a non-zero optimal noise level for the best FID rather than pure ODE sampling. And 20 steps is still 20 forward passes of a large network; getting to one or two requires distillation, or rebuilding the paths to be straighter, as in flow matching.

References

  1. Song, J., Meng, C., & Ermon, S. Denoising Diffusion Implicit Models. ICLR 2021.
  2. Ho, J., Jain, A., & Abbeel, P. Denoising Diffusion Probabilistic Models. NeurIPS 2020.
  3. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021.
  4. Karras, T., Aittala, M., Aila, T., & Laine, S. Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2022.
  5. Mokady, R., Hertz, A., Aberman, K., Pritch, Y., & Cohen-Or, D. Null-text Inversion for Editing Real Images using Guided Diffusion Models. CVPR 2023.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.