Adam
An optimizer is the rule that turns gradients into weight updates. Backpropagation tells you the direction of steepest descent for every parameter; the optimizer decides how far to step and how much to trust the raw gradient versus the history of gradients it has already seen. Everything about how fast a model trains, whether it converges at all, and how well it generalizes is downstream of this one choice. The whole field has converged on a small family of update rules, and understanding what each one does to the gradient is enough to reason about almost any training run.
Stochastic gradient descent is the baseline: step downhill by the gradient, scaled by the learning rate. Because the gradient is estimated on a mini-batch rather than the full dataset, the path is noisy — but that noise is a feature, acting as a regularizer that often helps generalization. Plain SGD is cheap in memory (no extra state) and still produces the best final accuracy on many vision benchmarks, at the cost of careful learning-rate tuning and slow progress through ravines in the loss surface.
Momentum fixes SGD's zig-zagging by accumulating a velocity. Instead of stepping by the current gradient, you keep an exponentially-decayed running average of past gradients and step by that. This damps the oscillation across a narrow valley and accelerates progress along its floor, the way a heavy ball rolls through small bumps. It is the single most cost-effective upgrade to SGD and costs just one extra copy of the parameters.
Adaptive methods give every parameter its own learning rate. RMSProp scales each update by a running average of that parameter's squared gradients, so frequently-updated weights take smaller steps and rarely-updated ones take larger steps. Adam combines the two ideas — it tracks a first moment (momentum) and a second moment (RMSProp-style variance), applies a bias correction so early steps are not too small, and has become the default optimizer for essentially all transformer training. Its price is memory: it stores two extra values per parameter, which for a large model is a substantial share of the training footprint.
AdamW is the version you actually want for large models. The original Adam folds weight decay into the gradient, which interacts badly with the adaptive scaling; AdamW *decouples* weight decay and applies it directly to the weights, which measurably improves generalization and is now the standard recipe for training LLMs. Newer optimizers such as Lion push further on memory efficiency by keeping only a sign-based momentum term, trading a little quality for a smaller optimizer state.
| Optimizer | Extra state / param | Adaptive per-param LR | Note | Typical use |
|---|---|---|---|---|
| SGD | none | No | Noisy but generalizes well | Vision, fine-tuning |
| SGD + momentum | 1x | No | Damps oscillation, accelerates | CNNs, ResNets |
| RMSProp | 1x | Yes | Per-parameter scaling | RNNs, RL |
| Adam | 2x | Yes | Momentum + variance + bias fix | Default for transformers |
| AdamW | 2x | Yes | Decoupled weight decay | LLM pretraining |
The instinct is to treat the optimizer as a hyperparameter you inherit from whatever tutorial you started with — "use AdamW, it works." It is more useful to see each optimizer as a specific policy for spending the gradient: SGD trusts the raw noisy gradient, momentum trusts a smoothed history of it, and Adam reshapes it per-parameter using both the average and the variance it has observed. That reshaping is what buys robustness to bad learning rates, and its cost is the extra state you have to hold in memory. Read an optimizer through a how-it-reshapes-the-raw-gradient lens rather than a which-one-converges-fastest lens, and choices like SGD-for-vision, AdamW-for-LLMs, and Lion-when-memory-is-tight stop being lore and become a straight trade between robustness and the memory you can afford.