Loss Landscape Smoothness

Loss Landscape Smoothness refers to the geometric properties of the loss function surface in parameter space — smooth landscapes (low curvature, wide minima) correlate with better generalization, while rough landscapes (sharp minima, high curvature) correlate with poor generalization.

Smoothness Metrics

• Hessian Eigenvalues: The eigenvalues of the loss Hessian measure local curvature — smaller eigenvalues = smoother.
• Sharpness: The maximum loss change within a neighborhood of the minimum — sharp minima generalize poorly.
• Filter Normalization: Visualize the loss landscape by plotting loss along random directions, normalized by filter norms.
• PAC-Bayes: Sharpness-aware generalization bounds relate the width of minima to generalization error.

Why It Matters

• Generalization: Models converging to flat minima generalize better — SAM (Sharpness-Aware Minimization) explicitly seeks flat minima.
• Batch Size: Large batch sizes tend to find sharp minima — small batches explore more and find flatter minima.
• Architecture: Skip connections (ResNets) create smoother loss landscapes — one reason they train more easily.

Loss Landscape Smoothness is the geometry of good solutions — flatter, smoother loss landscapes produce models that generalize better.