Novograd

NovoGrad is an adaptive optimizer that uses layer-wise second moments instead of per-parameter moments — dramatically reducing optimizer memory while maintaining competitive training performance, especially for NLP and speech models.

How Does NovoGrad Work?

• Layer-Wise Second Moment: $v_l = \beta_2 v_l + (1-\beta_2) g_l ^2$ (one scalar per layer, not per parameter).
• Normalized Gradient: $\hat{g}_l = g_l / \sqrt{v_l}$ (normalize by layer-wise second moment).
• Momentum: Standard first-moment EMA on the normalized gradient.
• Paper: Ginsburg et al. (2019).

Why It Matters

• Memory Savings: One scalar per layer vs. one value per parameter -> massive memory reduction for the second moment buffer.
• Speech/NLP: Designed for and effective on Jasper (speech) and BERT (NLP) training.
• Large Models: Memory savings enable larger models or batch sizes within the same GPU memory.

NovoGrad is the frugal adaptive optimizer — achieving Adam-like adaptation with a fraction of the memory by thinking in layers instead of parameters.