Score-Based Generative Models
Score-Based Generative Models are a class of generative models that learn the score function ∇_x log p(x)—the gradient of the log-probability density with respect to the data—rather than the density itself, then use the learned score to generate samples through iterative score-based sampling procedures such as Langevin dynamics. This approach avoids the normalization constant computation that makes direct density modeling intractable for complex, high-dimensional distributions.
Why Score-Based Generative Models Matter in AI/ML:
Score-based models provide state-of-the-art generative quality by sidestepping the fundamental challenge of normalizing constant computation, leveraging the fact that the score function contains all the information needed for sampling without requiring a tractable partition function.
• Score function — The score ∇_x log p(x) is a vector field pointing in the direction of increasing log-density at every point in data space; following this gradient (with noise) from any starting point converges to samples from p(x) via Langevin dynamics
• Score matching training — Directly minimizing E[ s_θ(x) - ∇_x log p(x) ²] is intractable (requires knowing the true score); denoising score matching instead trains on noisy data: s_θ(x̃) ≈ ∇_{x̃} log p(x̃|x) = -(x̃-x)/σ², which is tractable and consistent
• Multi-scale noise perturbation — Score estimation is inaccurate in low-density regions (few training examples); adding noise at multiple scales (σ₁ > σ₂ > ... > σ_N) fills in low-density regions and creates a sequence of score functions from coarse to fine
• Connection to diffusion — Score-based models and denoising diffusion probabilistic models (DDPMs) are equivalent formulations: the DDPM denoiser ε_θ is related to the score by s_θ(x_t, t) = -ε_θ(x_t, t)/σ_t; this unification bridges the two research communities
• SDE formulation — Song et al. unified score-based and diffusion models through stochastic differential equations (SDEs): the forward SDE gradually adds noise, and the reverse-time SDE (requiring the score function) generates samples by denoising
| Component | Role | Implementation | ||
|---|---|---|---|---|
| Score Network s_θ | Estimates ∇_x log p(x) | U-Net, Transformer (time-conditioned) | ||
| Noise Schedule | Multi-scale perturbation | σ₁ > σ₂ > ... > σ_N or continuous σ(t) | ||
| Training Loss | Denoising score matching | E[ | s_θ(x+σε) + ε/σ | ²] |
| Sampling | Reverse-time SDE/ODE | Langevin dynamics, predictor-corrector | ||
| SDE Forward | dx = f(x,t)dt + g(t)dw | VP-SDE, VE-SDE, sub-VP-SDE | ||
| SDE Reverse | dx = [f - g²∇log p]dt + gdw̄ | Score-guided denoising |
Score-based generative models represent a paradigm shift in generative modeling by learning the gradient of the log-density rather than the density itself, unifying with diffusion models through the SDE framework and achieving state-of-the-art image generation quality by sidestepping normalization constant computation while enabling flexible, iterative sampling through learned score functions.