Adafactor

Adafactor is a memory-efficient adaptive optimizer designed for training large models — replacing Adam's per-parameter second moment buffer with a factored approximation, reducing optimizer memory from $O(mn)$ to $O(m + n)$ for each matrix parameter.

How Does Adafactor Work?

• Factored Second Moments: For a weight matrix $W \in \mathbb{R}^{m \times n}$, instead of storing the full $m \times n$ second moment, store row factors ($m$) and column factors ($n$).
• Reconstruction: $v_{ij} \approx r_i \cdot c_j / \bar{r}$ (outer product approximation).
• No Momentum: Optionally omits first moment (momentum) to save more memory.
• Paper: Shazeer & Stern (2018).

Why It Matters

• Memory Savings: For large transformer models (billions of parameters), Adafactor saves 30-50% optimizer memory vs. Adam.
• T5: Used to train Google's T5 model family (11B parameters).
• Large Models: Essential when model size pushes against GPU memory limits.

Adafactor is Adam on a memory diet — achieving comparable optimization quality with dramatically less memory through smart factorization.