Diffusion-Lm

Diffusion-LM is a language model that applies continuous diffusion to word embeddings for controllable text generation — mapping discrete tokens to continuous embedding vectors, applying Gaussian diffusion in embedding space, and rounding back to discrete tokens, enabling plug-and-play controllable generation.

Diffusion-LM Architecture

• Embedding: Map discrete tokens to continuous embedding vectors — $e(w) \in \mathbb{R}^d$.
• Forward Diffusion: Add Gaussian noise to embedding sequence — gradually corrupt the embeddings.
• Reverse Denoising: Learn to denoise embeddings — predict clean embeddings from noisy ones.
• Rounding: Map denoised continuous embeddings back to discrete tokens using nearest-neighbor lookup.

Why It Matters

• Controllability: Diffusion enables gradient-based control — guide generation toward desired attributes (topic, sentiment, syntax) via classifier guidance.
• Non-Autoregressive: Generates all positions simultaneously — enables global planning and coherent generation.
• Flexibility: Plug-and-play classifiers can control any attribute without retraining the base model.

Diffusion-LM is diffusion meets language — applying continuous diffusion in embedding space for flexible, controllable text generation.