Normalizing Flow Generative

Normalizing Flows are the generative model family that learns an invertible transformation between a simple base distribution (e.g., standard Gaussian) and a complex target distribution (e.g., natural images) — where the invertibility enables exact likelihood computation via the change-of-variables formula, and the transformation is composed of learnable invertible layers (coupling layers, autoregressive transforms, continuous flows) that progressively reshape the simple distribution into the complex data distribution.

Mathematical Foundation

If z ~ p_z(z) is the base distribution and x = f(z) is the invertible transformation, the data distribution is:
p_x(x) = p_z(f⁻¹(x)) × |det(∂f⁻¹/∂x)|

The Jacobian determinant accounts for how the transformation stretches or compresses probability density. For the transformation to be practical:
1. f must be invertible (bijective).
2. The Jacobian determinant must be efficient to compute (not O(D³) for D-dimensional data).

Coupling Layer Architectures

RealNVP / Glow:
- Split input into two halves: x = [x_a, x_b].
- Transform: y_a = x_a (identity), y_b = x_b ⊙ exp(s(x_a)) + t(x_a).
- s() and t() are arbitrary neural networks (no invertibility requirement — they parameterize the transform, not perform it).
- Jacobian is triangular → determinant is the product of diagonal elements (O(D) instead of O(D³)).
- Inverse: x_b = (y_b - t(x_a)) ⊙ exp(-s(x_a)), x_a = y_a. Exact inversion!
- Stack multiple coupling layers, alternating which half is transformed.

Autoregressive Flows (MAF, IAF):
- Transform each dimension conditioned on all previous dimensions: x_i = z_i × exp(s_i(x_{<i})) + t_i(x_{<i}).
- More expressive than coupling layers (full autoregressive conditioning) but sequential in one direction.
- MAF: slow sampling (sequential), fast density evaluation. IAF: fast sampling, slow density evaluation.

Continuous Normalizing Flows (CNF)

Instead of discrete invertible layers, define a continuous ODE:
dz/dt = f_θ(z(t), t)

The time-evolution from t=0 (noise) to t=1 (data) defines the transformation. The log-determinant of the Jacobian is computed via the trace of the Jacobian (Hutchinson's trace estimator) integrated over time. Solved with adaptive ODE solvers (Neural ODE).

Flow Matching (Lipman et al., 2023)

A simplified, simulation-free training method for CNFs:
- Define a probability path from noise to data using conditional optimal transport.
- Train a vector field v_θ(x, t) to match the target flow using a simple regression loss.
- No ODE solving during training — just one network evaluation per training step. 10-100× faster training than neural ODE-based flows.
- Flow matching has become a key component in modern generative models (Stable Diffusion 3 uses it for image generation).

Advantages vs. Other Generative Models

Normalizing Flows are the mathematically principled generative model that provides exact density computation through invertible transformations — and through flow matching, have become a practical, efficient alternative to diffusion models for high-quality generation.