Pipeline Parallel

Pipeline parallelism distributes model layers across multiple GPUs as sequential stages, with microbatching to maintain high utilization by keeping multiple mini-batches in flight simultaneously, reducing the "bubble" overhead of sequential pipeline execution. The concept: split model into k stages on k GPUs; each GPU processes one stage and passes activations to the next. Without microbatching, GPU i waits idle while later stages process, creating large "bubbles." Microbatching: divide batch into m microbatches; as soon as GPU 1 finishes microbatch 1, it starts microbatch 2 while GPU 2 processes microbatch 1. This keeps pipeline filled. GPipe: seminal approach with synchronous microbatching; bubble overhead = (k-1)/(m+k-1), approaching 0 as microbatches increase. PipeDream: asynchronous pipeline with weight stashing, reducing bubble but requiring extra memory for weight versions. Memory trade-offs: pipeline parallel reduces memory per GPU (only one stage's parameters) but requires activation storage (or recomputation) between forward and backward passes. Combining with other parallelism: often used with data parallelism (replicate pipeline) and tensor parallelism (within stages) for large-scale training. Pipeline parallelism enables training models too large for single GPU memory while maintaining reasonable hardware utilization.