Sequence Parallelism

Sequence parallelism distributes the sequence dimension of activations across GPUs, reducing per-GPU memory consumption for long-context LLM training and enabling context lengths that wouldn't fit on a single device. Problem: transformer activations scale as O(batch × sequence × hidden_dim)—for long sequences (32K-1M+ tokens), activation memory becomes the bottleneck even when model weights are distributed via tensor parallelism. Sequence parallelism approaches: (1) Megatron-SP—split non-tensor-parallel operations (LayerNorm, Dropout) along sequence dimension; (2) DeepSpeed Ulysses—partition sequence across GPUs, use all-to-all communication for attention; (3) Ring Attention—distribute sequence in ring topology, overlap communication with computation. Megatron-SP (Korthikanti et al., 2022): in tensor parallel regions, activations are already split across GPUs. For non-TP operations (LayerNorm, Dropout), Megatron-SP splits along sequence dimension and uses all-gather/reduce-scatter (replacing the all-reduce in standard TP). Benefit: reduces activation memory by TP factor for these operations. DeepSpeed Ulysses: each GPU holds sequence_length/N tokens for all attention heads. Before attention, all-to-all gathers full sequence for each head subset. After attention, all-to-all redistributes. Communication cost: O(N²) all-to-all messages. Best with fast NVLink. Ring Attention: sequence divided into chunks distributed across GPUs in a ring. Each GPU computes attention for its local query chunk against key/value blocks passed around the ring. Overlaps communication with computation. Scales to very long sequences (1M+ tokens). Memory savings: sequence parallelism across P GPUs reduces per-GPU activation memory by ~P×. Enables training with context lengths otherwise impossible. Combinations: sequence parallelism typically combined with tensor, pipeline, and data parallelism for maximum efficiency on large models with long contexts.