Adabelief

AdaBelief is an adaptive optimizer that adapts the learning rate based on the "belief" in the current gradient direction — using the deviation of the gradient from the expected gradient (EMA), rather than the gradient magnitude itself, as the adaptive scaling factor.

How Does AdaBelief Work?

• Key Change: Instead of $v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$ (Adam), use $v_t = \beta_2 v_{t-1} + (1-\beta_2)(g_t - m_t)^2$ (AdaBelief).
• Interpretation: If the gradient $g_t$ matches the momentum $m_t$ (strong belief), take a large step. If they diverge (weak belief), take a small step.
• Effect: Adapts to gradient predictability, not just magnitude.
• Paper: Zhuang et al. (2020).

Why It Matters

• Fast Convergence: Combines the fast convergence of Adam with the generalization of SGD.
• Better Generalization: Outperforms Adam on test accuracy while maintaining fast training.
• Stability: Less likely to diverge on noisy gradients compared to Adam.

AdaBelief is the confidence-weighted optimizer — stepping boldly when gradients are predictable and cautiously when they're erratic.