Energy-Based Model

Energy-Based Models (EBMs) are a class of generative models that define a probability distribution through an energy function — $p_\theta(x) = \exp(-E_\theta(x)) / Z$ where lower energy corresponds to higher probability, and the model learns to assign low energy to data-like inputs.

Key Concepts

• Energy Function: $E_\theta(x)$ is a neural network mapping inputs to a scalar energy value.
• Partition Function: $Z = \int \exp(-E_\theta(x)) dx$ — intractable normalization constant.
• Sampling: MCMC methods (Langevin dynamics, HMC) generate samples by following the energy gradient.
• Training: Contrastive divergence, score matching, or noise contrastive estimation (NCE) avoid computing $Z$.

Why It Matters

• Flexibility: EBMs can model arbitrary distributions without architectural constraints (no decoder, no normalizing flow).
• Composability: Multiple EBMs can be combined by adding energies — $E_{joint} = E_1 + E_2$.
• Discriminative + Generative: The same energy function can be used for both classification and generation (JEM).

EBMs are learning an energy landscape — defining probability through energy where likely configurations sit in low-energy valleys.