Gradient Normalization

Gradient Normalization is the practice of normalizing gradient magnitudes during training — either by clipping the gradient norm to a maximum value (gradient clipping) or by scaling gradients to have unit norm, preventing exploding gradients and stabilizing training.

Types of Gradient Normalization

• Gradient Clipping by Norm: $\hat{g} = g \cdot \min(1, c/ g )$. Clips when $ g > c$.
• Gradient Clipping by Value: Clip each element independently: $\hat{g}_i = \text{clip}(g_i, -c, c)$.
• Unit Norm: Scale to unit norm: $\hat{g} = g / g $.
• Gradient Scaling: Scale gradients by a constant factor (used in mixed-precision training).

Why It Matters

• Stability: Prevents exploding gradients in RNNs, transformers, and deep networks.
• Necessary for LLMs: Gradient clipping (typically $c = 1.0$) is standard in all transformer pre-training.
• Mixed Precision: Loss scaling + gradient unscaling is critical for FP16/BF16 training.

Gradient Normalization is the safety valve for deep learning — preventing gradient explosions that would otherwise crash training.