Sophia

Sophia (Second-order Clipped Stochastic Optimization) is a lightweight second-order optimizer designed for large language model pre-training — using a diagonal Hessian estimate to provide per-parameter curvature information, achieving 2x faster training than Adam with minimal overhead.

How Does Sophia Work?

• Hessian Estimate: Uses a diagonal Gauss-Newton approximation or Hutchinson estimator to estimate per-parameter curvature $h_t$.
• Update: $\theta_{t+1} = \theta_t - eta \cdot \text{clip}(m_t / \max(h_t, \gamma), 1)$ (gradient divided by curvature, clipped).
• Clipping: The clipping prevents excessively large steps in flat directions (where $h_t \approx 0$).
• Cost: ~15% overhead over Adam (Hessian estimated every 10-20 steps).

Why It Matters

• 2x Speedup: Reaches the same loss as Adam in half the training steps for LLMs (GPT-2, LLaMA).
• Curvature-Aware: Steps more aggressively in flat directions and conservatively in sharp ones — optimal for the heterogeneous loss landscape of LLMs.
• Scalable: Designed for billion-parameter models.

Sophia is the curvature-aware optimizer for LLMs — using lightweight second-order information to navigate the loss landscape twice as fast as Adam.