Gradient Compression

Gradient Compression is a distributed training optimization technique that reduces the communication volume of gradients — sending only the most important gradient information between workers, cutting communication overhead by 100-1000x at the cost of a small approximation.

The Communication Bottleneck

• AllReduce of gradients: Must communicate all parameters each step.
• GPT-3 (175B params): 175B × 4 bytes = 700GB per AllReduce step.
• Inter-node bandwidth: 100Gbps = 12.5 GB/s → 56 seconds per step.
• Solution: Reduce what's communicated without hurting convergence.

Top-K Sparsification

• Gradient vector: Most values are small, few are large.
• Top-K: Communicate only the K largest (by magnitude) gradient elements.
• K = 0.1%: 1000x compression — only 0.1% of gradients transmitted.
• Error feedback: Accumulate skipped gradients locally → include in next step.
• Without error feedback: Top-K diverges. With it: Convergence preserved.

PowerSGD (2019)

• Low-rank approximation: G \approx PQ^T where P, Q are low-rank factors.
• Compress gradient matrix G (m×n) to P (m×r) + Q (n×r), r << \min(m,n).
• Rank-4 PowerSGD: 16x compression with minimal accuracy loss.
• Default optimizer option in PyTorch DDP.

1-bit SGD / SignSGD

• Extreme compression: Communicate only sign of gradient (1 bit per element).
• 32x compression vs. FP32.
• QSGD: Stochastic quantization to k bits — adjustable compression ratio.

Communication Overlap

• Combine compression with overlap: Compute layer N+1 while communicating layer N gradients.
• Bucket allreduce: Group small layers into buckets — amortize communication overhead.

Convergence Guarantees

• With error feedback: Top-K and PowerSGD converge to same quality as uncompressed SGD.
• Trade-off: Compression ratio vs. wall-clock speedup vs. accuracy degradation.

Gradient compression is a key technique for scaling distributed training beyond NVLink speed — when training across multiple nodes connected by slower Ethernet or InfiniBand, compression can save 50-200K in compute costs for large model training runs.