Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Classifier-Free Guidance: Buying Prompt Adherence with a Second Forward Pass

6 minute read

Published:

TL;DR: A conditional diffusion model trained the obvious way treats the prompt as a weak hint. Classifier guidance sharpened it, but needed a separate classifier trained on noisy images. Classifier-free guidance removes that classifier by training a single network on both the conditional and unconditional task — drop the condition on roughly 10% of examples — and then extrapolating past the conditional prediction at sampling time. The guidance weight \(w\) is a dial from "diverse and vague" to "on-prompt and samey", and turning it up means you are no longer sampling from the true conditional distribution.

Conditioning is not control

Feeding a text embedding into the denoiser gives you \(\epsilon_\theta(\mathbf{x}_t, t, c)\), a model of \(p(\mathbf{x}\mid c)\). Trained this way it works, but weakly: the prompt nudges the sample rather than governing it. The reason is that the training loss rewards covering the data, and the conditional and unconditional distributions of natural images overlap heavily. A model can score well while largely ignoring \(c\).

What users want is not a sample from \(p(\mathbf{x}\mid c)\). They want a sample from the part of \(p(\mathbf{x}\mid c)\) where the prompt is unambiguously satisfied — a sharpened, mode-seeking version of it.

Classifier guidance, and why it was awkward

Dhariwal and Nichol’s answer was to attach an explicit classifier \(p_\phi(c \mid \mathbf{x}_t)\) and push the sample uphill on it. Because \(\boldsymbol{\epsilon}_\theta = -\sqrt{1-\bar{\alpha}_t}\,\nabla_{\mathbf{x}_t}\log p_t(\mathbf{x}_t)\), Bayes’ rule in score space becomes a simple additive correction on the predicted noise:

\[ \hat{\boldsymbol{\epsilon}} = \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t) \;-\; s\,\sqrt{1-\bar{\alpha}_t}\;\nabla_{\mathbf{x}_t}\log p_\phi(c \mid \mathbf{x}_t) \]

Here \(s\) is the guidance scale and \(\bar{\alpha}_t\) the cumulative signal coefficient of the forward process. The correction says: move in the direction that increases the classifier’s confidence in \(c\).

The awkwardness is the classifier itself. It has to accept \(\mathbf{x}_t\), not a clean image, so an off-the-shelf ImageNet classifier is useless — you must train a bespoke one on noisy inputs across all timesteps. That is a second model, a second training pipeline and a second failure mode. Worse, gradients of a classifier on near-pure noise are exactly the adversarial-example regime: the classifier can be made confident by perturbations that do not correspond to real image structure. And for free-form text there is no classifier to train in the first place.

The trick: be your own classifier

Ho and Salimans noticed the correction never needed a classifier explicitly. Applying Bayes’ rule the other way,

\[ \nabla_{\mathbf{x}_t}\log p(c \mid \mathbf{x}_t) = \nabla_{\mathbf{x}_t}\log p(\mathbf{x}_t \mid c) - \nabla_{\mathbf{x}_t}\log p(\mathbf{x}_t) \]

so the classifier gradient is the difference between two scores the diffusion model can already produce, provided it can also run unconditionally. Getting that for free costs one line in the data loader: during training, replace \(c\) with a null token \(\varnothing\) with probability around 10%. One network, one loss, two behaviours.

At sampling time, substitute and rearrange, and the guided noise prediction is a straight-line extrapolation:

\[ \tilde{\boldsymbol{\epsilon}} = \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, \varnothing) \;+\; w\,\bigl(\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, c) - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, \varnothing)\bigr) \]

Read it as a lever. At \(w = 0\) you get the unconditional model and the prompt is ignored. At \(w = 1\) the two terms cancel to \(\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, c)\) — ordinary conditional sampling. At \(w > 1\) you overshoot past the conditional prediction, along the direction that distinguishes “with prompt” from “without prompt”. Typical text-to-image settings sit between 5 and 10. (Some papers write \(\tilde{\boldsymbol{\epsilon}} = (1+s)\boldsymbol{\epsilon}_c - s\,\boldsymbol{\epsilon}_\varnothing\); that is the same equation with \(w = 1+s\).)

Guidance as extrapolation along the conditional–unconditional differenceA horizontal axis in noise-prediction space. A point at the left is labelled epsilon with the null condition, at guidance weight 0. A point in the middle is labelled epsilon conditioned on c, at guidance weight 1. The arrow continues past it to a third point labelled guided epsilon at weight 7.5, which lies six and a half difference-lengths beyond the conditional point. A bracket under the first two points is labelled "the difference the prompt makes", and a bracket under the extrapolated region is labelled "extrapolation: off-distribution". w = 0 w = 1 w = 7.5 ε(x, ∅) ε(x, c) guided ε the difference the prompt makes extrapolation — no data ever lived here Noise-prediction space at a fixed timestep
Notice that the useful setting is not between the two model outputs but far beyond one of them. Guidance is extrapolation, not interpolation, which is exactly why it degrades sample statistics as \(w\) grows.
Key Insight — what distribution you are actually sampling: converting the guided prediction back into a score gives \(\tilde{s} = \nabla\log p(\mathbf{x}) + w\,\nabla\log p(c\mid\mathbf{x})\), which is the score of \(p(\mathbf{x})\,p(c\mid\mathbf{x})^{w}\), not of \(p(\mathbf{x}\mid c)\). For \(w>1\) the likelihood term is raised to a power greater than one, so the density is sharpened around whatever the model finds most prompt-like. Guidance is deliberate, controlled bias. It is also not exact even as a sampler for that tilted density, because the sequence of tilted marginals is not the forward diffusion of any single distribution.

The trade-off, stated honestly

Guidance weightPrompt adherenceDiversityTypical artefacts
\(w = 1\)loose; prompt often partially ignoredhighestwashed-out, generic compositions
\(w \approx 3\)reliable on the main subjectgoodmild
\(w \approx 7\)strong, including attributesnoticeably reducedrising saturation and contrast
\(w \ge 15\)over-literalcollapses towards one compositionblown-out colours, cartoon edges, clipping

Two costs are unavoidable. The first is compute: every sampling step needs both a conditional and an unconditional forward pass, so sampling is about twice as expensive per step (batching the two halves together hides the latency, not the FLOPs). The second is the saturation blow-up. Extrapolating in \(\boldsymbol{\epsilon}\)-space pushes the implied clean image \(\hat{\mathbf{x}}_0\) outside the valid pixel range, and the errors compound over steps. Imagen’s dynamic thresholding — rescaling \(\hat{\mathbf{x}}_0\) by a per-step percentile rather than hard-clipping it — is the standard patch, and it is what makes very high guidance weights usable at all.

Where it breaks: guidance interacts with the sampler. At low step counts the extrapolated field is stiffer, so a weight that looks fine at 50 steps can produce burnt-out images at 10. And because guidance narrows the distribution, evaluating a guided model on diversity-sensitive metrics measures the guidance weight almost as much as it measures the model.

Guidance is why text-to-image diffusion works at all in practice, and it composes with everything else: it is orthogonal to where the diffusion happens (latent diffusion), to how the trajectory is integrated (samplers), and to whether the path is stochastic or straight (flow matching).

References

  1. Dhariwal, P., & Nichol, A. Diffusion Models Beat GANs on Image Synthesis. NeurIPS 2021.
  2. Ho, J., & Salimans, T. Classifier-Free Diffusion Guidance. NeurIPS 2021 Workshop on Deep Generative Models.
  3. Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. ICML 2022.
  4. Saharia, C., Chan, W., Saxena, S., et al. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. NeurIPS 2022.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.