Batch Size

Batch size is the number of examples processed together in one forward-backward pass before weight update. Trade-offs: Large batches: More stable gradients, GPU utilization, faster wall-clock (with parallelism), but may generalize worse. Small batches: Noisier gradients (regularization effect), less memory, possibly better generalization. Memory impact: Larger batch = more activation memory. Often the limiting factor for batch size. Learning rate scaling: Large batches often need higher learning rate. Linear scaling rule: double batch, double LR (with warmup). Gradient accumulation: Simulate large batches on limited memory by accumulating across steps. Effective batch size: Per-device batch x devices x accumulation steps. What matters for training dynamics. LLM training: Large batches (millions of tokens) for efficiency. Requires careful LR tuning. Critical batch size: Beyond some size, more compute without proportional improvement. Diminishing returns. Recommendations: Maximize batch size within memory, scale LR appropriately, use accumulation if needed. Hyperparameter: Often tuned alongside learning rate. Larger models may benefit from larger batches.