Multinomial Diffusion

Multinomial Diffusion is a discrete diffusion model where the forward process corrupts categorical data using a categorical (multinomial) noise distribution — at each timestep, each token has a probability of being replaced by any other token in the vocabulary according to a multinomial transition matrix.

Multinomial Diffusion Details

• Transition Matrix: $q(x_t | x_{t-1}) = Cat(x_t; Q_t x_{t-1})$ — categorical distribution over vocabulary.
• Uniform Noise: The simplest scheme transitions toward a uniform distribution over all tokens.
• Absorbing: Alternative scheme transitions toward a single [MASK] token — absorbing state diffusion.
• Reverse: $p_\theta(x_{t-1} | x_t) = Cat(x_{t-1}; pi_\theta(x_t, t))$ — neural network predicts clean token probabilities.

Why It Matters

• Natural Fit: Multinomial diffusion is mathematically natural for text, categorical features, and one-hot encoded data.
• D3PM: Structured Denoising Diffusion Models (Austin et al., 2021) formalized multinomial and absorbing diffusion.
• Flexibility: Different transition matrices enable different noise schedules — uniform, absorbing, or token-similarity-based.

Multinomial Diffusion is random token scrambling and unscrambling — a discrete diffusion process using categorical transitions for generating text, molecules, and other categorical data.