Gradient Noise

Gradient Noise is the deliberate addition of noise to gradient updates during training — typically Gaussian noise with decaying variance, which helps escape local minima, improves generalization, and can approximate Bayesian posterior sampling.

How Does Gradient Noise Work?

• Injection: $\tilde{g} = g + \mathcal{N}(0, \sigma_t^2)$ where $\sigma_t$ decays over training.
• Schedule: $\sigma_t = \sigma_0 / (1 + t)^\gamma$ with $gamma approx 0.55$.
• Mini-Batch Noise: SGD inherently has gradient noise from mini-batch sampling. Added noise amplifies this effect.
• Paper: Neelakantan et al., "Adding Gradient Noise Improves Learning" (2015).

Why It Matters

• Escape Local Minima: Noise helps SGD escape sharp local minima and find flatter ones (better generalization).
• Bayesian Connection: Gradient noise with appropriate scaling can approximate Langevin dynamics for Bayesian inference.
• Deep Networks: Particularly helpful for very deep networks where deterministic gradients can get trapped.

Gradient Noise is controlled randomness in optimization — deliberately shaking the optimizer to help it find better solutions in the loss landscape.