Gradient Clipping

Gradient Clipping — a technique that limits the magnitude of gradients during backpropagation to prevent exploding gradients from destabilizing training.

The Problem

• In deep networks (especially RNNs/Transformers), gradients can grow exponentially during backpropagation
• One bad batch → huge gradient → catastrophic weight update → model diverges (loss goes to NaN)

Methods

• Clip by Norm: Scale the entire gradient vector if its norm exceeds a threshold

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

If $ g > max\_norm$: $g \leftarrow g \times \frac{max\_norm}{ g }$
Preserves gradient direction, just limits magnitude

• Clip by Value: Clamp each gradient element independently to [-value, +value]

torch.nn.utils.clip_grad_value_(model.parameters(), clip_value=0.5)

Simpler but can change gradient direction

Common Settings

• Transformer training: `max_norm=1.0` (standard)
• RNN/LSTM training: `max_norm=5.0` (more aggressive needed)
• LLM training: `max_norm=1.0` (GPT, LLaMA, etc.)

When to Use

• Always for RNNs and Transformers
• When training with large learning rates
• When using mixed precision (FP16 gradients can overflow more easily)

Gradient clipping is a simple safety mechanism that virtually every modern deep learning training pipeline includes.