Gradient

Gradients and Backpropagation

What is Backpropagation?
Backpropagation computes gradients of the loss with respect to each parameter, enabling gradient-based optimization.

The Chain Rule
For a composition of functions $y = f(g(x))$:

Backprop applies this recursively through the network.

Forward and Backward Pass

Forward Pass
Compute outputs layer by layer, storing intermediate activations:

Input → Layer1 → (activations1) → Layer2 → (activations2) → ... → Loss

Backward Pass
Compute gradients layer by layer, from loss to inputs:

dLoss → dLayer_n → dLayer_{n-1} → ... → dLayer_1

Gradient Flow in Transformers

Key Components
| Component | Gradient Consideration |
|-----------|----------------------|
| Layer Norm | Stabilizes gradient magnitudes |
| Residual connections | Enable gradient flow to early layers |
| Attention | Gradients flow through softmax |
| FFN | Standard MLP gradients |

Residual Connections Are Critical

output = layer(x) + x  # Skip connection

# Gradient flows through both paths
d_output = d_layer + d_identity

Without residuals, gradients would vanish in deep networks.

Gradient Issues

Vanishing Gradients
- Gradients become too small in early layers
- Solutions: Residual connections, Layer Norm, careful initialization

Exploding Gradients
- Gradients become too large, causing instability
- Solutions: Gradient clipping, Layer Norm, lower learning rate

Gradient Clipping

# Clip gradient norm
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Memory for Gradients
Storing activations for backward pass is memory-intensive:
- Solution 1: Gradient checkpointing (recompute instead of store)
- Solution 2: Mixed precision (FP16/BF16 activations)
- Solution 3: Activation offloading to CPU

Monitoring Gradients

# Check gradient norms during training
for name, param in model.named_parameters():
    if param.grad is not None:
        print(f"{name}: {param.grad.norm():.4f}")