Autoregressive Diffusion

Autoregressive Diffusion is a hybrid generative model that combines autoregressive (left-to-right) generation with diffusion-based denoising — generating tokens sequentially but using a diffusion process at each position, or applying diffusion with an autoregressive ordering constraint.

Autoregressive Diffusion Variants

• ARDM (Autoregressive Diffusion Models): Generate tokens in a random order — each token is generated conditioned on previously generated tokens.
• Order-Agnostic: Learn to generate in ANY order, not just left-to-right — order is sampled during training.
• Upsampling: Generate a coarse sequence autoregressively, then refine with diffusion — hierarchical approach.
• Absorbing + AR: Combine absorbing diffusion (unmask one token at a time) with autoregressive conditioning.

Why It Matters

• Flexibility: Unlike pure AR models (fixed left-to-right), ARDM can generate in any order — more flexible decoding.
• Quality: Combining AR conditioning with diffusion can improve generation quality over pure non-autoregressive methods.
• Speed: Can decode faster than pure AR (generate multiple tokens per step) while maintaining coherence.

Autoregressive Diffusion is sequential denoising — combining the coherence of autoregressive generation with the flexibility and quality of diffusion models.