Flat Minima

Flat Minima are regions of the loss landscape where the loss remains low over a wide neighborhood of parameter values — characterized by small eigenvalues of the Hessian matrix, and empirically associated with better generalization performance.

What Are Flat Minima?

• Definition: A minimum where the loss changes slowly when parameters are perturbed -> wide valley.
• Hessian: Small eigenvalues of $H = \nabla^2 mathcal{L}$ indicate flatness.
• Measures: Sharpness (max eigenvalue), trace of Hessian, volume of the low-loss region.
• Connection: Flat minima have high posterior volume in Bayesian interpretation -> preferred by Occam's razor.

Why It Matters

• Generalization: Flat minima generalize better because the solution is robust to parameter perturbations (which approximate the distribution shift between train and test).
• SAM: Sharpness-Aware Minimization explicitly seeks flat minima by optimizing worst-case loss in a perturbation ball.
• Batch Size: Large batch SGD tends to find sharp minima; small batch SGD finds flatter ones.

Flat Minima are the wide valleys of good generalization — solutions where the model's performance is robust to small changes in its parameters.