Stochastic Weight Averaging

SWA (Stochastic Weight Averaging) is an optimization technique that averages multiple checkpoints collected during training with a high or cyclical learning rate — the averaged weights converge to wider, flatter minima that generalize better than the final checkpoint alone.

How Does SWA Work?

• Train: Train normally until near convergence.
• SWA Phase: Continue with a high or cyclic learning rate for additional epochs.
• Collect: Save the model weights at the end of each SWA epoch.
• Average: $\theta_{SWA} = \frac{1}{T}\sum_t \theta_t$ (running average of collected checkpoints).
• Paper: Izmailov et al. (2018).

Why It Matters

• Flat Minima: SWA finds wider minima that generalize better (loss landscape is flat around SWA solution).
• Free Improvement: 0.5-1.5% accuracy improvement with minimal additional training cost.
• PyTorch Built-In: Available as torch.optim.swa_utils.AveragedModel.

SWA is averaging your way to a better model — collecting checkpoints along a high-learning-rate trajectory to find wide, flat minima.