Global Batch

Global batch is the total number of samples contributing to one optimizer update across all devices and accumulation passes - it is the optimizer-facing batch size that determines gradient statistics and learning-rate scaling behavior.

What Is Global batch?

• Definition: Global batch aggregates local micro-batches from all parallel workers over accumulation steps.
• Optimization Link: Many hyperparameters, especially learning rate and warmup, depend on global batch.
• System Decoupling: Hardware topology may change while preserving the same global batch target.
• Measurement: Should be logged explicitly for every run to ensure comparable experiment interpretation.

Why Global batch Matters

• Convergence Consistency: Matching global batch helps maintain similar optimization dynamics across cluster sizes.
• Scaling Decisions: Global batch is the key anchor for linear scaling and large-batch experiments.
• Benchmark Fairness: Performance comparisons are misleading if global batch differs silently.
• Reproducibility: Exact batch semantics are required to recreate prior model quality outcomes.
• Cost Analysis: Batch size affects step count and runtime, directly influencing training economics.

How It Is Used in Practice

• Formula Tracking: Compute and log global batch from micro-batch, world size, and accumulation settings.
• Policy Coupling: Tie LR, momentum, and scheduler parameters to explicit global batch checkpoints.
• Scale Migration: When adding GPUs, rebalance micro-batch and accumulation to preserve intended global batch.

Global batch is the central quantity that connects distributed systems configuration to optimizer behavior - controlling it explicitly is required for reliable scaling and reproducibility.