Energy Based Model

Energy-Based Model (EBM) is a generative model that assigns a scalar energy to each configuration of variables — learning a function $E_\theta(x)$ such that low-energy states correspond to real data and high-energy states to unlikely configurations.

Core Concept

• Probability: $p_\theta(x) = \frac{\exp(-E_\theta(x))}{Z(\theta)}$
• $Z(\theta) = \int \exp(-E_\theta(x)) dx$ — partition function (intractable in general).
• Training: Push $E(x_{real})$ low, push $E(x_{fake})$ high.
• No explicit generative process required — just a scalar score function.

Training Challenges

• Computing $Z(\theta)$: Intractable for continuous high-dimensional data.
• Solution: Contrastive Divergence (CD): Replace exact gradient with approximate using MCMC samples.
• CD-k: Run MCMC for k steps from data points → approximate negative phase.

Restricted Boltzmann Machine (RBM)

• Bipartite graph: Visible units $v$ and hidden units $h$, no intra-layer connections.
• Energy: $E(v,h) = -v^T W h - b^T v - c^T h$
• Exact conditional distributions: $p(h|v)$ and $p(v|h)$ are factorial — efficient Gibbs sampling.
• Deep Belief Networks: Stack of RBMs — early deep learning (Hinton, 2006).

Modern EBMs

• JEM (Joint Energy-Based Model): EBM for both classification and generation.
• Score-based models: $\nabla_x \log p(x)$ (score function) — equivalent to EBM.
• Diffusion models: Can be viewed as hierarchical EBMs.

MCMC Sampling

• Stochastic Gradient Langevin Dynamics (SGLD): Sample from EBM by gradient descent + noise.
• $x_{t+1} = x_t - \alpha \nabla_x E_\theta(x_t) + \epsilon$, $\epsilon \sim N(0,I)$.

Applications

• Anomaly detection: Outliers have high energy.
• Data-efficient learning: EBMs learn compact energy landscape.
• Scientific applications: Molecule energy functions (MMFF, OpenMM).

Energy-based models are a unifying framework connecting Boltzmann machines, diffusion models, and score-based models — their elegant probabilistic formulation makes them particularly powerful for physics-inspired applications and anomaly detection where likelihood estimation matters.