Discrete Diffusion

Discrete Diffusion models are generative models that apply the diffusion framework to discrete data (tokens, categories, graphs) — instead of adding Gaussian noise to continuous values, discrete diffusion corrupts data by randomly replacing tokens with other tokens or a mask state, then learns to reverse this corruption process.

Discrete Diffusion Approach

• Forward Process: Gradually corrupt discrete tokens — replace with random tokens or [MASK] at increasing rates.
• Transition Matrix: A categorical transition matrix $Q_t$ defines the corruption probabilities at each timestep.
• Absorbing State: One variant uses an absorbing [MASK] state — tokens are progressively masked until all are masked.
• Reverse Process: A neural network learns to predict the original tokens from corrupted sequences.

Why It Matters

• Text Generation: Enables non-autoregressive text generation using diffusion — competitive with autoregressive models.
• Molecules: Discrete diffusion generates molecular graphs — atoms and bonds are discrete structures.
• Categorical Data: Natural for any domain with categorical variables — proteins, music, code.

Discrete Diffusion is noise-and-denoise for categories — extending the diffusion model framework from continuous data to discrete tokens and structures.