Dynamic Loss Scaling

Dynamic loss scaling is the adaptive method that adjusts loss scale during mixed-precision training to avoid overflow and underflow - it automates numeric stabilization for fp16 regimes where gradient magnitude varies over time.

What Is Dynamic loss scaling?

• Definition: Multiply loss by a scale factor before backward pass, then unscale gradients before optimizer step.
• Adaptive Logic: Decrease scale when overflow is detected and increase scale after stable intervals.
• Failure Handling: Optimizer step can be skipped on overflow to avoid corrupt parameter updates.
• Framework Support: Implemented in common mixed-precision toolchains for automated stability control.

Why Dynamic loss scaling Matters

• Numerical Safety: Protects small gradients from underflow and large gradients from overflow.
• Training Continuity: Automatic adjustment reduces manual tuning effort across model phases.
• FP16 Viability: Makes half-precision training practical for a wider range of architectures.
• Operational Robustness: Adapts to changing gradient distributions during long runs.
• Productivity: Reduces failed runs caused by precision instability.

How It Is Used in Practice

• Initial Scale: Start from a high but safe scale and let runtime controller adjust as needed.
• Overflow Detection: Check gradients for inf or nan before applying optimizer updates.
• Telemetry: Log scale value, skipped steps, and overflow events to guide precision debugging.

Dynamic loss scaling is a key stability mechanism for mixed-precision optimization - adaptive scaling keeps gradients in a representable range while preserving training performance.