Lamb

LAMB (Layer-wise Adaptive Moments optimizer for Batch training) is an optimizer specifically designed for large-batch distributed training — extending Adam with layer-wise trust ratios that normalize the update magnitude per layer, enabling stable training with batch sizes up to 65K or more.

How Does LAMB Work?

• Base: Standard Adam momentum and adaptive learning rate computation.
• Trust Ratio: Scale each layer's update by $\phi( w ) / Adam\_update $ (ratio of weight norm to update norm).
• Effect: Prevents any single layer from receiving disproportionately large or small updates.
• Paper: You et al. (2020).

Why It Matters

• Large Batch Training: Enables training BERT in 76 minutes (was 3 days with smaller batches).
• Scaling Efficiency: Near-linear scaling up to thousands of GPUs.
• Distributed Training: The go-to optimizer for large-scale distributed pre-training runs.

LAMB is the team coordinator for distributed training — ensuring that large-batch updates are balanced across layers for maximum training throughput.