Adam Optimizer

Adam optimizer combines momentum and adaptive learning rates, the default choice for most deep learning. Algorithm: Maintains exponential moving averages of gradient (m) and squared gradient (v). Update: w -= lr * m / (sqrt(v) + eps). Key features: Per-parameter learning rates adapt to gradient history. Momentum smooths updates. Bias correction for early steps. Hyperparameters: lr (learning rate, ~1e-4 to 3e-4 for LLMs), beta1 (momentum, 0.9), beta2 (squared gradient decay, 0.999), epsilon (stability, 1e-8). Variants: AdamW: Decouples weight decay from gradient update. Preferred for transformers. Adafactor: Memory-efficient, factorizes second moment. 8-bit Adam: Quantized states for memory savings. Memory cost: 2 states per parameter (m, v) plus parameters = 3x parameter memory. Comparison to SGD: Adam converges faster early, SGD may generalize better with tuning. Adam is default. For LLMs: AdamW with beta1=0.9, beta2=0.95 common. Higher beta2 for stability. Best practices: Use AdamW for transformers, tune learning rate first, default betas usually fine.