Diffusion Beyond Images: Inpainting, Audio, Video, Molecules, Robots
Published:
Inpainting: conditioning for free
Filling a masked region needs no new model. Because the forward process gives \(q(\mathbf{x}_t\mid\mathbf{x}_0)\) in closed form, the known pixels can be told what value to hold at every noise level. RePaint’s update replaces the known region at each reverse step, with \(\mathbf{m}=1\) on the region you are keeping:
The generated region sees a correctly-noised context at its own noise level, so it can match it. Done naively the seam is visible: the model conditions on the known part but the known part never conditions on the model, so global coherence lags. RePaint’s fix is resampling — re-noise \(\mathbf{x}_{t-1}\) back to \(\mathbf{x}_t\) and redo the step several times, letting information flow both ways before committing. The general lesson is that any constraint you can express as “these coordinates are known” costs nothing extra, and dedicated inpainting checkpoints exist mainly to improve on this baseline, not to enable it.
Super-resolution: the average is not an answer
A low-resolution image is consistent with many high-resolution ones, so the L2-optimal prediction is their mean — which is exactly the blur that regression-based super-resolution produces. Diffusion instead samples one of the consistent completions. Conditioning is usually just concatenation: upsample the low-resolution input to the target grid and stack it onto the model’s input channels. This also gives a clean route to high resolution by cascading, where a low-resolution base model is followed by two or three conditional upsamplers, each an easier problem than direct high-resolution synthesis. Noise conditioning augmentation — corrupting the low-resolution conditioning during training — is what stops each stage from breaking on the previous stage’s artefacts.
Audio: two places to put the noise
Waveform models such as DiffWave and WaveGrad diffuse the raw signal, conditioned on a mel-spectrogram. The attraction over autoregressive vocoders is parallelism: a waveform has tens of thousands of samples per second, and sample-by-sample generation is inherently sequential, while a diffusion step touches every sample at once. The alternative is to diffuse a time–frequency representation — a mel-spectrogram or a learned audio latent — and decode with a separate vocoder, which is the same two-stage argument as latent diffusion and is what text-to-audio systems generally use. Spectrogram diffusion is cheaper and easier to condition on text; waveform diffusion avoids the vocoder’s own error floor.
Video: consistency as a joint distribution
The hard part of video is not per-frame quality but temporal coherence, and the reason diffusion handles it is structural. Rather than generating frames one at a time and hoping they agree, video diffusion denoises the whole clip jointly: the noisy tensor has a time axis, and attention is factorised into spatial and temporal blocks so every frame’s denoising sees every other frame. Consistency then falls out of the model’s joint distribution instead of being imposed by an optical-flow post-process. The cost is memory that grows with clip length, which is why practical systems generate short clips and extend them by conditioning on the last frames — and why drift over long horizons remains unsolved.
Molecules and proteins: the noise must respect the symmetry
A molecule is a set of atoms with 3D coordinates, and its energy does not change if you rotate or translate it. A generative model should inherit that. Two things are needed. The denoiser must be E(3)-equivariant, so that rotating the input rotates the predicted noise identically. And the noise distribution itself must be compatible: isotropic Gaussian noise is rotation-invariant, which is convenient, but no probability density on \(\mathbb{R}^{3N}\) can be translation-invariant, because it would not be normalisable. Equivariant diffusion models therefore work in the linear subspace with zero centre of mass, projecting out the translation before adding noise. For protein backbones the state is a set of rigid frames, so diffusion runs jointly on positions in \(\mathbb{R}^3\) and orientations on \(\mathrm{SO}(3)\), which needs a diffusion process defined on the manifold rather than in a vector space.
Robot policies: multimodality is the point
Behaviour cloning with a mean-squared-error loss fails on tasks where two different actions are both correct — go left or right around an obstacle — because the average of two valid actions is often invalid. Diffusion Policy models the conditional distribution over a short chunk of future actions given recent observations, and samples from it. Two properties do the work: the multimodal distribution is represented rather than averaged away, and generating a chunk rather than a single action gives temporal consistency without an explicit smoother. The action dimension is small, so the usual sampling-cost objection is much weaker here than it is for images — though closed-loop control still pushes practitioners towards few-step samplers.
| Modality | What is diffused | Why diffusion fits |
|---|---|---|
| Inpainting | image (masked) | forward process gives free conditioning on known pixels |
| Super-resolution | high-res image | one-to-many; regression returns a blurred mean |
| Audio | waveform or spectrogram | parallel over time, unlike autoregressive vocoders |
| Video | full clip tensor | coherence emerges from a joint distribution |
| Molecules | 3D coordinates, frames | noise can be made symmetry-compatible |
| Robot actions | action chunk | multiple valid actions must not be averaged |
References
- Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., & Van Gool, L. RePaint: Inpainting using Denoising Diffusion Probabilistic Models. CVPR 2022.
- Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M. Image Super-Resolution via Iterative Refinement. IEEE TPAMI 2022.
- Kong, Z., Ping, W., Huang, J., Zhao, K., & Catanzaro, B. DiffWave: A Versatile Diffusion Model for Audio Synthesis. ICLR 2021.
- Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., & Fleet, D. J. Video Diffusion Models. NeurIPS 2022.
- Hoogeboom, E., Satorras, V. G., Vignac, C., & Welling, M. Equivariant Diffusion for Molecule Generation in 3D. ICML 2022.
- Watson, J. L., Juergens, D., Bennett, N. R., et al. De novo design of protein structure and function with RFdiffusion. Nature 2023.
- Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., & Song, S. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.