Diffusion Model
Diffusion Models are generative models that learn to reverse a gradual noising process, transforming pure Gaussian noise back into structured data through iterative denoising steps — producing state-of-the-art image, audio, and video generation quality that has surpassed GANs, powering systems like Stable Diffusion, DALL-E 3, Midjourney, and Sora.
Forward Process (Adding Noise)
- Start with a clean data sample x₀ (e.g., an image).
- At each timestep t, add a small amount of Gaussian noise: $x_t = \sqrt{\alpha_t} \cdot x_{t-1} + \sqrt{1 - \alpha_t} \cdot \epsilon$.
- After T steps (T ≈ 1000): x_T ≈ pure Gaussian noise.
- This process requires no learning — it's a fixed schedule.
Reverse Process (Denoising — The Learned Part)
- A neural network (typically a U-Net or Transformer) learns to predict the noise ε added at each step.
- Starting from pure noise x_T, iteratively denoise: $x_{t-1} = \frac{1}{\sqrt{\alpha_t}}(x_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_\theta(x_t, t)) + \sigma_t z$.
- After T reverse steps → generates a clean sample from the learned distribution.
Training Objective
- Simple MSE loss: $L = E_{t, x_0, \epsilon}[ \epsilon - \epsilon_\theta(x_t, t) ^2]$.
- Sample a random timestep t, add noise to get x_t, predict the noise, minimize error.
- No adversarial training, no mode collapse — stable optimization.
Key Variants
| Model | Innovation | Speed |
|---|---|---|
| DDPM (Ho et al. 2020) | Original formulation | Slow (1000 steps) |
| DDIM | Deterministic sampling, fewer steps | 10-50 steps |
| Latent Diffusion (LDM) | Diffuse in VAE latent space, not pixel space | Fast (Stable Diffusion) |
| Flow Matching | Straighter ODE paths | 1-10 steps possible |
| Consistency Models | Direct single-step generation | 1-2 steps |
Conditioning and Guidance
- Text conditioning: Text encoder (CLIP/T5) provides embedding → cross-attention in U-Net.
- Classifier-Free Guidance (CFG): $\epsilon_{guided} = \epsilon_{uncond} + w \cdot (\epsilon_{cond} - \epsilon_{uncond})$.
- Scale w = 7-15 for high-quality, text-aligned generation.
- ControlNet: Additional conditioning on edges, depth maps, poses.
Latent Diffusion (Stable Diffusion Architecture)
- VAE encodes 512×512 image → 64×64 latent representation (8x compression).
- Diffusion operates in latent space → 64x less computation than pixel-space diffusion.
- U-Net with cross-attention for text conditioning.
- VAE decoder converts denoised latent back to pixel image.
Diffusion models are the dominant generative paradigm as of 2024-2025 — their combination of training stability, output quality, and flexible conditioning has made them the foundation of commercial image generation, video synthesis, drug design, and audio generation systems.