Communication-Efficient Training

Communication-Efficient Training encompasses the set of techniques to reduce the communication overhead in distributed deep learning — addressing the key bottleneck where gradient synchronization between workers dominates training time.

Communication Reduction Strategies

• Gradient Compression: Sparsification (top-K, random) and quantization (1-bit, ternary) reduce message size.
• Local SGD: Workers perform multiple local gradient steps before synchronizing — reduce communication frequency.
• Gradient Accumulation: Accumulate gradients over multiple mini-batches before communicating.
• Decentralized: Replace the central parameter server with peer-to-peer gossip communication.

Why It Matters

• Scalability: Communication cost grows with number of workers — communication efficiency enables scaling to more GPUs.
• Network Bottleneck: In datacenter training, network bandwidth is 100-1000× slower than compute — communication dominates.
• Edge/Federated: In federated learning, communication is extremely expensive (slow WAN links) — efficiency is critical.

Communication-Efficient Training is maximizing compute-per-byte — reducing the communication needed to synchronize distributed training without sacrificing model quality.