← Home
A First-Principles Tutorial · Chapter 06

Flow Models

Invertible transformations, exact likelihoods, and the modern convergence of flows with diffusion.

By Prashant Kulkarni · with help from Claude OpusCC BY-NC 4.0
1 The core idea

Invertible transformations and exact likelihood

Every generative model family answers the same question differently: how do you turn simple noise into complex data? Normalizing flows take the mathematically cleanest path — define an invertible mapping between a Gaussian and the data distribution, then compute the exact likelihood using the change of variables formula. No lower bounds, no adversaries, no denoising.

A normalizing flow defines a bijection $f_\theta: \mathcal{Z} \to \mathcal{X}$. To sample: draw $z \sim \mathcal{N}(0,I)$, compute $x = f_\theta(z)$. To evaluate density: compute $z = f_\theta^{-1}(x)$, apply the change of variables formula. Both directions work because $f_\theta$ is invertible.

Invertible chain: sample forward, evaluate backward z ~ N(0,I) simple base sample → ← evaluate f₁ f₂ ··· f_K x ~ p_data complex data K INVERTIBLE LAYERS Every layer must be invertible with a tractable Jacobian determinant
Forward (green): sample from Gaussian, push through K invertible layers to produce data. Inverse (red): take data, pull back to Gaussian for exact density evaluation. Each layer must have a tractable Jacobian determinant.Original SVG
Why exact likelihood matters

Exact likelihood gives flows unique advantages: principled model comparison (log-likelihood on held-out data), no mode collapse (maximum likelihood covers all modes), no posterior gap (unlike VAEs), and lossless compression. The price: every layer must be invertible, constraining architecture design.

2 The math

The change of variables formula

$$\log p_\theta(x) = \log p_\mathcal{Z}\!\left(f_\theta^{-1}(x)\right) + \log\left|\det \frac{\partial f_\theta^{-1}}{\partial x}\right|$$

Two terms: the base density at the mapped point, plus a volume correction from the Jacobian determinant. When $f$ stretches a region, density decreases to keep total probability at 1. The Jacobian determinant measures exactly how much local volume changes.

For a chain of $K$ layers: $\log p_\theta(x) = \log p_\mathcal{Z}(z_0) - \sum_{k=1}^K \log|\det J_{f_k}|$. Training maximizes average log-likelihood — a clean, unbiased objective with no variational bounds or surrogate losses.

The design constraint: a general $D \times D$ determinant costs $O(D^3)$. For images ($D = 196{,}608$ for 256×256×3), this is impossible. Every practical flow architecture uses special structure — triangular Jacobians, block-diagonal patterns — to bring the cost to $O(D)$.

3 Architecture

Coupling layers — the trick that makes the Jacobian tractable

The affine coupling layer (Dinh et al., 2015/2017) splits the input into two halves and transforms one conditioned on the other:

$$x_a = z_a \qquad x_b = z_b \odot \exp(s(z_a)) + t(z_a)$$

where $s$ (scale) and $t$ (translation) are arbitrary neural networks. They can be deep ResNets — their complexity doesn't affect the Jacobian cost.

Affine coupling: split, condition, transform — triangular Jacobian z_a z_b identity (unchanged) s(z_a) scale t(z_a) shift x_b = z_b·exp(s)+t x_a x_b log|det J| = Σ s(z_a)_j — just a sum! O(D) cost
Because $x_a$ doesn't depend on $z_b$, the Jacobian is lower triangular. Its determinant is simply the product of diagonal entries — the sum of scale values. Inversion is equally cheap: $z_b = (x_b - t(x_a)) \odot \exp(-s(x_a))$.Original SVG · concept after Dinh et al. (2017)

Since each layer only transforms half the dimensions, you alternate which half is transformed across layers. Stack 8–16 coupling layers with alternating masks and the flow becomes expressive enough for complex distributions.

4 Autoregressive flows

The MAF vs IAF tradeoff

Autoregressive flows take coupling to its extreme: each dimension conditions on all previous dimensions. This maximizes expressiveness per layer but introduces a fundamental asymmetry:

MAF

Conditions on observed data. Density evaluation is fast (parallel). Sampling is slow ($D$ sequential steps). Use for: training, evaluation.

IAF

Conditions on latent variables. Sampling is fast (parallel). Density evaluation is slow. Use for: generation, VAE posteriors.

MAF and IAF are mathematical inverses. Coupling layers are a special case with two groups — sacrificing expressiveness for speed in both directions.

5 Going continuous

Continuous normalizing flows — infinite layers

Take $K \to \infty$ with infinitesimal steps: instead of a discrete chain, define the transformation as an ODE: $dx/dt = v_\theta(x, t)$. The velocity field $v_\theta$ is any neural network — no invertibility constraints. To sample, integrate from $t=0$ to $t=1$. FFJORD (Grathwohl et al., 2019) showed this works, using Hutchinson's trace estimator for the log-determinant. Downside: ODE simulation during training is expensive. This sets the stage for flow matching.

6 Milestone

Glow — 1×1 convolutions

Glow (Kingma & Dhariwal, 2018) was the first flow to generate realistic faces, using three innovations: invertible 1×1 convolutions as learned permutations (replacing fixed channel shuffling), multi-scale architecture (split off dimensions at each scale), and actnorm (data-dependent activation normalization). Glow demonstrated that flows could reach near-GAN quality — with exact log-likelihood as a bonus.

7 The modern approach

Flow matching — simulation-free training

Flow matching (Lipman et al., 2023) trains a continuous normalizing flow without ODE simulation and without architectural constraints. Define a linear interpolation between noise and data:

$$x_t = (1-t)\,x_0 + t\,x_1, \qquad x_0 \sim \mathcal{N}(0,I), \quad x_1 \sim p_\text{data}$$

The conditional velocity along this path is simply $x_1 - x_0$ — the direction from noise to data. The training objective is regression on this velocity:

$$\mathcal{L}_\text{CFM} = \mathbb{E}_{t,\,x_0,\,x_1}\!\left[\|v_\theta(x_t, t) - (x_1 - x_0)\|^2\right]$$
Flow matching: four-step training loop, structurally identical to DDPM ① SAMPLEx₁~data, x₀~N(0,I), t~U ② INTERPOLATEx_t = (1-t)·x₀ + t·x₁ ③ PREDICTv̂ = v_θ(x_t, t) ④ LOSS‖v̂ − (x₁−x₀)‖² gradient update → repeat vs DDPM: target is velocity (x₁−x₀) not noise (ε) · interpolation is linear not variance-preserving
The flow matching training loop is structurally identical to DDPM: sample, interpolate, predict, compute loss. The differences — linear interpolation, velocity target — produce straighter sampling trajectories requiring fewer ODE steps.Original SVG · concept after Lipman et al. (2023)
Why flow matching is taking over

The linear interpolation is the optimal transport path — the straightest possible trajectory between noise and data. Straighter = fewer ODE solver steps at inference (10–50 instead of 20–1000). Stable Diffusion 3, Flux, and SD 3.5 all use rectified flow matching.

8 Straightening

Rectified flows — even straighter paths

Rectified flows (Liu et al., 2023) iteratively straighten trajectories: train a flow, generate matched pairs $(x_0, x_1)$ by running the ODE, retrain on straight-line interpolation between matched pairs, repeat. After rectification, trajectories are nearly straight — solvable with 1–2 Euler steps. This is the foundation of SD3's fast generation.

9 Landscape

Classical flows vs flow matching vs diffusion

Three generations of flow-based generative models CLASSICAL FLOWS RealNVP, Glow, MAF ✓ exact likelihood ✓ fast sampling ✗ constrained architecture ✗ limited quality 2015–2020 FLOW MATCHING ★ CFM, rectified flows ✓ no arch constraints ✓ straight trajectories ✓ few sampling steps ✓ simple training 2023–now DIFFUSION DDPM, score SDE ✓ no arch constraints ✓ high quality + diversity ✗ curved trajectories ✗ many sampling steps 2020–2023 Flow matching and diffusion have converged — SD3 and Flux blur the boundary
Classical flows had exact likelihood but constrained architectures. Diffusion removed constraints but used curved paths. Flow matching achieves unconstrained architectures with straight OT paths — the current state of the art.Original SVG

The deep connection: $v_t(x) = f(t)x + g(t)^2 \nabla_x \log p_t(x)$ — velocity is a linear combination of drift and score. Flow matching and diffusion use the same architectures (DiT), same conditioning (CLIP/T5), same infrastructure. Whether SD3 is "a diffusion model" or "a flow model" has no clean answer.

10 Field notes

Flows on 2D distributions

Source

Experiments from the Module 06 notebook. Implements RealNVP coupling layers and flow matching on 2D distributions (moons, rings).

Watching coupling layers warp space

The notebook visualizes how 8 affine coupling layers progressively warp a Gaussian grid into the two-moons distribution. Each layer applies a smooth, invertible deformation — grid lines bend but never cross (invertibility means no two points map to the same location). Early layers do coarse rearrangement; later layers handle fine-grained shape matching.

Exact density on a grid

Because flows compute exact likelihood, you can evaluate $\log p_\theta(x)$ at every point on a 2D grid and plot the learned density surface. The result matches the empirical distribution — peaks at the moons, valleys between. No other generative model family gives you this exact, pointwise density evaluation.

Flow matching trajectories

The flow matching implementation shows particle trajectories from noise to data. Unlike coupling layers (discrete jumps), flow matching particles follow nearly straight lines — each particle takes an almost direct path from noise to data. This straightness directly explains why flow matching needs fewer ODE steps.

Velocity field at multiple timesteps

At $t=0$, the velocity arrows point outward from the origin toward data. At $t=0.5$, the field is most structured — the target shape is visible. At $t \approx 1$, arrows become small — particles have arrived.

11 Playground

Watch particles flow from noise to data

Particles start as Gaussian noise and follow straight trajectories to form a ring of clusters. Drag the slider to scrub through time, or press play. Toggle "trails" to see how straight the paths are.

t = 0.00 · Gaussian noise · drag slider or press play

⁂ ⁂ ⁂
12 Closing

Takeaways & exercises

  1. Flows compute exact likelihood via the change of variables formula — no bounds, no adversaries.
  2. Coupling layers make the Jacobian tractable by exploiting triangular structure — $O(D)$ not $O(D^3)$.
  3. MAF and IAF are inverses — fast density/slow sampling vs fast sampling/slow density.
  4. Flow matching removes all constraints — learn a velocity field via regression, no invertibility needed.
  5. Straight trajectories = fewer steps. OT interpolation produces straighter paths than diffusion.
  6. Flow matching and diffusion have converged. SD3 and Flux use flow matching with diffusion-style architectures.
Exercises

1. Why must flow transformations be invertible? What would go wrong if two inputs mapped to the same latent code?

2. Write out the 4×4 Jacobian for a 2+2 affine coupling split. Verify it's triangular and its determinant is $\prod \exp(s_j)$.

3. Open the notebook. Compare coupling-layer grid warping with flow matching trajectories. Which looks straighter?

4. In the playground above, toggle trails on. Are the paths straight? Compare mentally to the diffusion demo in Chapter 04.

5. List three differences between flow matching and DDPM training (interpolation, target, path shape).

Further reading

  • Dinh et al. (2015/2017) — NICE and RealNVP. Coupling-layer flows.
  • Papamakarios et al. (2017) — MAF. Masked autoregressive flows.
  • Kingma & Dhariwal (2018) — Glow. Invertible 1×1 convolutions.
  • Grathwohl et al. (2019) — FFJORD. Continuous normalizing flows.
  • Lipman et al. (2023) — Flow Matching for Generative Modeling.
  • Liu et al. (2023) — Rectified Flows. Iterative straightening.
  • Esser et al. (2024) — Stable Diffusion 3. Flow matching in production.
  • P. Kulkarni — Module 06 notebook.