Data Parallelism

Data parallelism replicates the model on each device and processes different data batches in parallel. How it works: Copy complete model to each GPU, each processes different mini-batch, average gradients across devices, update weights synchronously. Gradient synchronization: All-reduce operation aggregates gradients across devices. Communication overhead scales with parameter count. Scaling: Effective batch size = per-device batch size x number of devices. More devices = larger effective batch. Advantages: Simple to implement, near-linear speedup for compute-bound training, well-supported in frameworks. Limitations: Each device must fit entire model in memory. Doesnt help if model too large for single GPU. Communication bottleneck: Gradient sync can become bottleneck at scale. Gradient compression, async methods help. Implementation: PyTorch DDP (DistributedDataParallel), Horovod, DeepSpeed ZeRO (hybrid). Best practices: Tune batch size with learning rate (linear scaling rule), use gradient accumulation for larger effective batch. Combination: Often combined with other parallelism strategies for large models (e.g., ZeRO, pipeline parallelism).