Linear Scaling Rule

Linear scaling rule is the heuristic that increases learning rate proportionally with batch-size growth - it is a common starting point for large-batch training but must be validated with stability controls.

What Is Linear scaling rule?

• Definition: If global batch is multiplied by k, initial learning rate is multiplied by k as first-order adjustment.
• Intuition: Larger batches reduce gradient noise, allowing larger optimization step sizes in many regimes.
• Applicability: Works best within bounded scaling ranges and with suitable optimizer settings.
• Failure Cases: At extreme batch sizes, linear scaling can destabilize training or hurt final quality.

Why Linear scaling rule Matters

• Practical Baseline: Provides simple, widely used initialization rule for distributed scaling experiments.
• Tuning Efficiency: Reduces search space when moving from single-node to multi-node batch sizes.
• Speed Potential: Correctly scaled LR can preserve convergence speed at higher throughput.
• Knowledge Transfer: Rule offers shared language across teams for scaling discussions.
• Optimization Discipline: Encourages structured rather than arbitrary hyperparameter adjustment.

How It Is Used in Practice

• Baseline Establishment: Start from well-performing small-batch configuration and apply proportional LR scaling.
• Warmup Integration: Use gradual LR ramp-up to avoid early divergence at high effective step sizes.
• Validation Sweep: Run narrow LR sweeps around linear target and choose by time-to-quality outcome.

Linear scaling rule is a useful first-order guide for large-batch optimization - it accelerates tuning, but robust results still require empirical validation and stability safeguards.