Radam

RAdam (Rectified Adam) is an adaptive optimizer that addresses the high variance of Adam's adaptive learning rate in early training — by computing the variance of the adaptive rate and automatically switching from SGD-like behavior (when variance is high) to full Adam (when variance is low).

How Does RAdam Work?

• Problem with Adam: In early training, the exponential moving average of squared gradients ($v_t$) is estimated from few samples -> high variance -> erratic learning rate.
• Solution: RAdam computes the variance of the adaptive rate. If variance exceeds a threshold, it falls back to momentum SGD.
• Automatic Warmup: Effectively provides automatic learning rate warmup without a warmup schedule.
• Paper: Liu et al. (2020).

Why It Matters

• No Warmup Needed: Eliminates the need for manual learning rate warmup schedules.
• Robust: More stable than Adam in the critical early training phase.
• Drop-In: Same API as Adam with no additional hyperparameters.

RAdam is Adam with self-correcting early training — automatically providing the warmup behavior that Adam needs but doesn't have.