Gradient Accumulation

Gradient Accumulation is a critical memory optimization technique universally employed in large-scale Vision Transformer and LLM training that mathematically simulates the effect of enormous batch sizes — often 4,096 or higher — on consumer or mid-range GPUs by splitting a single logical optimization step across multiple sequential forward-backward passes, accumulating the gradient contributions before executing a single weight update.

The Large Batch Requirement

• The ViT Convergence Mandate: Empirical research (DeiT, ViT-B/16) demonstrates that Vision Transformers require effective batch sizes of $1,024$ to $4,096$ to achieve reported accuracy. Smaller batch sizes produce noisy, high-variance gradient estimates that prevent the Self-Attention layers from learning stable, global feature representations.
• The Hardware Reality: A ViT-B/16 model processing a batch of $4,096$ images at $224 \times 224$ resolution simultaneously requires approximately $64$ GB of GPU memory for activations alone. A single NVIDIA A100 (40GB) or consumer RTX 4090 (24GB) physically cannot fit this batch.

The Accumulation Protocol

Gradient Accumulation resolves this by fragmenting the logical batch across time:
1. Micro-Batch Forward Pass: Process a small micro-batch of $B_{micro} = 32$ images through the full forward pass.
2. Backward Pass: Compute the gradients for this micro-batch. Crucially, do NOT update the weights.
3. Accumulate: Add the computed gradients to a running gradient accumulator buffer.
4. Repeat: Execute steps 1-3 a total of $K = 128$ times (the accumulation steps).
5. Update: After all $K$ micro-batches, divide the accumulated gradients by $K$ to compute the average, then execute a single optimizer step (AdamW weight update).

The effective batch size becomes $B_{effective} = B_{micro} \times K = 32 \times 128 = 4096$.

Mathematical Equivalence

Gradient accumulation produces mathematically identical gradients to true large-batch training under standard loss averaging. The gradient of the mean loss over $N$ samples is the mean of the per-sample gradients regardless of whether they are computed simultaneously or sequentially. The only difference is wall-clock time — accumulation processes the micro-batches serially rather than in parallel.

The Trade-Off

The technique trades approximately $30\%$ additional wall-clock training time (due to serial micro-batch processing) for a $50\%$ to $70\%$ reduction in peak GPU memory consumption, enabling the training of billion-parameter models on hardware that would otherwise be insufficient.

Gradient Accumulation is installment-plan optimization — paying the computational cost of a massive batch size in small, affordable sequential installments while receiving the mathematically identical gradient signal that a single enormous parallel computation would produce.