Flow Matching: Training a Velocity Field Without Ever Solving an ODE

Flows you can sample from but cannot afford to train

A continuous normalising flow (CNF) defines a sample as the endpoint of an ODE. Start from noise \(x_0 \sim p_0 = \mathcal{N}(\mathbf{0}, \mathbf{I})\) and push it along a learned velocity field \(v_\theta\):

Here \(\psi_t\) is the flow map — where a starting point has moved to by time \(t\) — and the model distribution is the law of \(\psi_1(x_0)\). The appeal is that \(v_\theta\) is unconstrained: any network defines a valid invertible map, with none of the triangular-Jacobian gymnastics of discrete normalising flows.

The problem is training it. Fitting by maximum likelihood needs the density, which follows the instantaneous change-of-variables formula

where \(\nabla \cdot v_\theta = \operatorname{tr}(\partial v_\theta / \partial x)\) is the divergence. One log-likelihood therefore costs a full ODE solve plus a divergence estimate at every solver step — an exact trace needs \(D\) backward passes in \(D\) dimensions, and the usual Hutchinson estimator is cheaper but noisier. The solver sits inside the training loop.

Regress the velocity instead

Suppose someone handed you a probability path \(p_t\) interpolating from noise at \(t=0\) to the data distribution \(q\) at \(t=1\), together with a vector field \(u_t\) that generates it. Then training would be a regression problem:

No solver, no divergence — one forward pass per sample. The catch is that neither \(p_t\) nor \(u_t\) is available: both are defined by the unknown data distribution.

Conditioning makes the target computable

The move that rescues this is to define the path per training example. Choose a conditional path \(p_t(x \mid x_1)\) that starts at \(\mathcal{N}(\mathbf{0}, \mathbf{I})\) at \(t=0\) and concentrates on the data point \(x_1\) at \(t=1\). Marginalising over the dataset recovers a path with the right endpoints:

The field generating this marginal path is, by the continuity equation, a posterior-weighted average of the conditional fields:

That integral runs over the whole data distribution, so it is hopeless to evaluate. But we never need to. The conditional flow matching objective drops the marginal entirely:

Read the expectation as a recipe: draw \(t \sim \mathcal{U}[0,1]\) and a data point \(x_1 \sim q\), draw \(x_t \sim p_t(\cdot \mid x_1)\), and regress \(v_\theta(x_t,t)\) onto the closed-form conditional velocity \(u_t(x_t \mid x_1)\). Every term comes from one example.

Why the swap is legal

Expand both squared norms. The quadratic term is the same in each, because averaging \(p_t(x\mid x_1)q(x_1)\) over \(x_1\) is exactly \(p_t(x)\):

The cross term is where the definition of \(u_t\) earns its keep. Substituting it, the awkward \(p_t(x)\) in the denominator cancels against the \(p_t(x)\) from the outer expectation:

which is precisely the cross term of \(\mathcal{L}_{\text{CFM}}\). The only surviving difference is \(\lVert u_t(x)\rVert^2\) against \(\lVert u_t(x\mid x_1)\rVert^2\), and neither depends on \(\theta\). So \(\nabla_\theta \mathcal{L}_{\text{FM}} = \nabla_\theta \mathcal{L}_{\text{CFM}}\): the two objectives differ by a constant and have the same minimiser.

The Gaussian probability path

The standard family takes each conditional path Gaussian, \(p_t(x \mid x_1) = \mathcal{N}\!\left(x;\, \mu_t(x_1),\, \sigma_t(x_1)^2 \mathbf{I}\right)\), with \(\mu_0 = \mathbf{0}, \sigma_0 = 1\) at the noise end and \(\mu_1 = x_1, \sigma_1 = \sigma_{\min}\) at the data end. The flow map is then an affine rescaling of the noise, \(x_t = \sigma_t x_0 + \mu_t\). Differentiating gives \(\dot{x}_t = \sigma'_t x_0 + \mu'_t\), and substituting \(x_0 = (x_t - \mu_t)/\sigma_t\) puts it in terms of the current position:

Two lines of calculus, and the target is closed-form for any schedule.

Diffusion is one path among many

Now the payoff. The conditional trajectory \(x_t = \sigma_t x_0 + \mu_t\) is a straight line in \(t\) exactly when both \(\mu_t\) and \(\sigma_t\) are affine in \(t\). Variance-preserving diffusion is not: read in this notation its schedule gives \(\mu_t = \alpha_t x_1\) and \(\sigma_t = \sqrt{1 - \alpha_t^2}\), and neither is affine, so the trajectory bends. Choosing \(\mu_t = t\,x_1\) and \(\sigma_t = 1 - (1-\sigma_{\min})t\) instead yields

a constant velocity along each conditional path — the rectified flow target. Here is the contrast in one dimension, with \(x_0 = -1\) and \(x_1 = 2\), against a variance-preserving path \(x_t = \cos(\tfrac{\pi t}{2})x_0 + \sin(\tfrac{\pi t}{2})x_1\):

Same endpoints, same framework, different geometry. The variance-preserving speed swings from 3.14 up to 3.50 and down to 1.57 — that variation is curvature, and curvature is what an ODE solver has to resolve with extra steps.

Sampling

Training never touched a solver. Sampling is where one finally appears. Draw \(x_0 \sim \mathcal{N}(\mathbf{0},\mathbf{I})\) and integrate

with any solver — Euler, midpoint, adaptive Runge–Kutta. The step count is now a deployment choice, decoupled from training, as it is for DDIM, whose deterministic sampler is the same idea reached from the score-based SDE direction. What flow matching adds is the freedom to design the path so that fewer steps suffice, which rectified flow takes to its conclusion. For how the two formulations line up term by term, see diffusion vs flow matching; for the corruption-based view they generalise, the diffusion overview.

References

1. Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., & Le, M. Flow Matching for Generative Modeling. ICLR 2023.
2. Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. Neural Ordinary Differential Equations. NeurIPS 2018.
3. Albergo, M. S., & Vanden-Eijnden, E. Building Normalizing Flows with Stochastic Interpolants. ICLR 2023.
4. Liu, X., Gong, C., & Liu, Q. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ICLR 2023.