Autoregressive Flows

Autoregressive Flows are a class of normalizing flow models that construct invertible transformations using autoregressive structure, where each output dimension depends only on the previous dimensions through a triangular Jacobian matrix. This autoregressive constraint enables exact and efficient computation of both the forward transformation and its log-determinant Jacobian, making density evaluation and sampling tractable while maintaining the expressiveness to model complex distributions.

Why Autoregressive Flows Matter in AI/ML:
Autoregressive flows provide exact density evaluation with flexible, learnable transformations, enabling precise likelihood computation for generative modeling, variational inference, and density estimation tasks where approximate methods are insufficient.

• Triangular Jacobian — The autoregressive structure produces a lower-triangular Jacobian matrix whose determinant is simply the product of diagonal elements: log|det J| = Σ log|∂y_i/∂x_i|; this O(d) computation replaces the general O(d³) determinant, making flows practical for high dimensions
• Masked Autoregressive Flow (MAF) — Each layer transforms x_i → y_i = x_i · exp(s_i(x_{<i})) + t_i(x_{<i}) where s and t are neural networks conditioned on previous dimensions; density evaluation is parallel (one forward pass) but sampling is sequential (autoregressive)
• Inverse Autoregressive Flow (IAF) — Reverses MAF's computational tradeoff: y_i = x_i · exp(s_i(y_{<i})) + t_i(y_{<i}); sampling is parallel (one pass) but density evaluation is sequential, making IAF ideal for variational inference where sampling speed matters
• Affine coupling layers — A simpler variant (RealNVP) splits dimensions into two groups: y_{1:d} = x_{1:d} (identity) and y_{d+1:D} = x_{d+1:D} · exp(s(x_{1:d})) + t(x_{1:d}); both forward and inverse are O(1) but less expressive per layer
• Stacking for expressiveness — Individual autoregressive layers have limited expressiveness; stacking multiple layers with permuted dimension orderings builds complex, multi-modal distributions while maintaining exact invertibility and tractable Jacobians

Autoregressive flows are the foundational architecture for normalizing flow models, providing exact, tractable density evaluation through the elegant constraint of triangular Jacobian structure that transforms complex distribution modeling into a sequence of simple, invertible transformations composable into arbitrarily expressive density estimators.