Sharpness-Aware Minimization

SAM (Sharpness-Aware Minimization) is an optimization technique that simultaneously minimizes both the loss value and the loss sharpness — by seeking parameters that lie in flat regions of the loss landscape where the worst-case loss within a perturbation neighborhood is minimized.

How Does SAM Work?

• Objective: $\min_\theta \max_{ \epsilon \leq \rho} \mathcal{L}(\theta + \epsilon)$ (minimize worst-case loss in a ball of radius $\rho$).
• Inner Step: Compute the adversarial perturbation: $\hat{\epsilon} = \rho \cdot \nabla_\theta \mathcal{L} / \nabla_\theta \mathcal{L} $.
• Outer Step: Compute gradient at $\theta + \hat{\epsilon}$ and use it to update $\theta$.
• Cost: 2x forward-backward passes per step (double the compute of standard SGD).

Why It Matters

• Generalization: Consistently improves test accuracy across architectures and datasets (0.5-2% on ImageNet).
• Simple: Drop-in replacement for standard optimizers with minimal code changes.
• Theory: Directly optimizes for flat minima, connecting the flatness-generalization hypothesis to a practical algorithm.

SAM is the optimizer that seeks wide valleys — trading compute for generalization by explicitly avoiding the sharp, brittle minima that lead to overfitting.