Asam

ASAM (Adaptive Sharpness-Aware Minimization) is an improvement over SAM that uses adaptive perturbation sizes for each parameter — normalizing the perturbation by the parameter magnitude, making SAM scale-invariant and more effective across different network architectures.

How Does ASAM Differ from SAM?

• SAM: Uses a uniform perturbation ball -> biased toward large-magnitude parameters.
• ASAM: $hat{\epsilon}_i = \rho cdot |w_i| cdot g_i / w odot g $ (perturbation proportional to parameter magnitude).
• Scale Invariance: Adaptive normalization ensures that the sharpness measure is invariant to parameter rescaling.
• Paper: Kwon et al. (2021).

Why It Matters

• Better Generalization: ASAM consistently outperforms SAM by 0.1-0.5% on ImageNet across architectures.
• Robustness: Less sensitive to the perturbation radius $\rho$ hyperparameter.
• Theory: Addresses the known theoretical limitation of SAM's non-adaptive perturbation.

ASAM is SAM with proportional fairness — ensuring that each parameter gets a perturbation sized appropriately for its scale, improving both theory and practice.