Score Matching for Ebms

Score Matching is a training method for energy-based models that avoids computing the intractable partition function — by matching the gradient (score) of the model's log-density to the gradient of the data distribution, which does not require normalization.

How Score Matching Works

• Score: The score function is $s_\theta(x) = \nabla_x \log p_\theta(x) = -\nabla_x E_\theta(x)$ (gradient of energy).
• Objective: Minimize $\mathbb{E}_{p_{data}}[|s_\theta(x) - \nabla_x \log p_{data}(x)|^2]$.
• Integration by Parts: The unknown $\nabla_x \log p_{data}$ can be eliminated, giving: $\mathbb{E}_{p_{data}}[\text{tr}(\nabla_x s_\theta) + \frac{1}{2}|s_\theta|^2]$.
• Denoising Score Matching: An equivalent objective that matches the score of the noise-perturbed distribution.

Why It Matters

• No Partition Function: Score matching completely avoids the intractable normalization problem.
• Diffusion Models: Modern diffusion models (DDPM, SDE-based) are trained with denoising score matching.
• Theoretically Sound: Score matching is consistent — the optimal model has the correct data score.

Score Matching is learning gradients instead of densities — training EBMs by matching the direction of steepest probability increase without computing $Z$.