Joint Energy-Based Models

JEM (Joint Energy-Based Models) is an approach that reinterprets a standard classifier as an energy-based model — the logit outputs of a classification network define an energy function $E(x) = -\text{LogSumExp}(f_\theta(x))$, enabling simultaneous discriminative classification and generative modeling from a single network.

How JEM Works

• Classifier: A standard neural network produces class logits $f_\theta(x) = [f_1(x), \ldots, f_K(x)]$.
• Energy: $E(x) = -\text{LogSumExp}_{y}(f_y(x))$ — the negative log-sum-exp of logits defines the energy.
• Classification: $p(y|x) = \text{softmax}(f_\theta(x))$ — standard discriminative classification.
• Generation: $p(x) propto \exp(-E(x))$ — sample using SGLD (Stochastic Gradient Langevin Dynamics).

Why It Matters

• Dual Use: One model does both classification AND generation — no separate generative model needed.
• Calibration: JEM-trained classifiers are better calibrated than standard classifiers.
• OOD Detection: The energy function naturally detects out-of-distribution inputs (high energy = OOD).

JEM is the classifier that generates — reinterpreting any classifier as a generative energy model for free.