Exponential Moving Average

EMA (Exponential Moving Average) is an optimization technique that maintains a shadow copy of model weights as an exponentially weighted moving average — the EMA model is used for evaluation/inference while the original model is used for gradient-based training.

How Does EMA Work?

• Update: After each training step: $\theta_{EMA} = \alpha \cdot \theta_{EMA} + (1-\alpha) \cdot \theta_{train}$ (typically $\alpha = 0.999$ or $0.9999$).
• Train: The main model $\theta_{train}$ is updated by the optimizer normally.
• Evaluate: Use $\theta_{EMA}$ for validation, testing, and deployment.
• Smooth: EMA averages out the noise from individual gradient updates.

Why It Matters

• Standard Practice: EMA is used in virtually all modern training recipes (ViT, diffusion models, LLMs).
• Free Accuracy: Typically 0.3-1.0% accuracy improvement at no additional training cost.
• Stability: The EMA model is more stable and less susceptible to overfitting than the raw model.

EMA is the smooth shadow model — maintaining a running average of weights that captures the model's best state throughout training.