Polyak Averaging

Polyak Averaging (Polyak-Ruppert Averaging) is a convergence acceleration technique that averages all parameter iterates during optimization — the average of all weights encountered during SGD converges faster than the final iterate for convex problems.

How Does Polyak Averaging Work?

• Iterate: Run SGD normally to get $\theta_1, \theta_2, ..., \theta_T$.
• Average: $\bar{\theta}_T = \frac{1}{T}\sum_{t=1}^T \theta_t$ (or use a tail average for non-convex).
• Theory: For convex problems, $\bar{\theta}_T$ converges at the optimal $O(1/T)$ rate even with a constant learning rate.
• Papers: Polyak (1990), Ruppert (1988).

Why It Matters

• Theoretical Foundation: Provides the theoretical justification for SWA and EMA techniques.
• Constant Learning Rate: Enables using a larger, constant learning rate (the averaging cancels the noise).
• Practical: EMA is the modern, practical version of Polyak averaging with exponential forgetting.

Polyak Averaging is the theoretical foundation for weight averaging — the mathematically proven principle that averaging iterates accelerates convergence.