Swa-Gaussian

SWAG (SWA-Gaussian) is an approximation to Bayesian deep learning that uses the SWA trajectory to fit a Gaussian distribution over weights — capturing both the mean (SWA solution) and the covariance (spread of the SWA trajectory) for uncertainty estimation.

How Does SWAG Work?

• Mean: $\bar{\theta}$ from SWA (average of collected checkpoints).
• Covariance: Estimate a low-rank + diagonal covariance from the deviations of collected checkpoints from the mean.
• Posterior: $q(\theta) = \mathcal{N}(\bar{\theta}, \Sigma_{SWAG})$ (Gaussian approximate posterior).
• Inference: Sample multiple models from the posterior and average predictions (Bayesian model averaging).
• Paper: Maddox et al. (2019).

Why It Matters

• Uncertainty: Provides calibrated uncertainty estimates without the cost of full Bayesian inference.
• Efficient: Only requires the SWA trajectory — no special modifications to training.
• Scalable: Works for modern deep networks (ResNets, etc.) where full Bayesian methods are intractable.

SWAG is SWA with uncertainty — using the natural variation in the SWA trajectory to estimate a Bayesian posterior for calibrated predictions.