Gradient Sparsification

Gradient Sparsification is a communication reduction technique for distributed training that transmits only a subset of gradient components — sending only the most important (largest) gradients and accumulating the rest locally, reducing communication by 100-1000× with minimal accuracy loss.

Gradient Sparsification Methods

• Top-K: Send only the K largest gradient components by magnitude — deterministic selection.
• Random-K: Randomly sample K gradient components — stochastic, unbiased estimator.
• Threshold: Send only gradients exceeding a magnitude threshold — adaptive sparsity.
• Error Feedback: Accumulate unsent gradients locally and add them to the next round — prevents information loss.

Why It Matters

• Communication Bottleneck: In distributed training, gradient communication is often the bottleneck — sparsification eliminates it.
• 99%+ Sparsity: Deep learning gradients are often very sparse — sending only 0.1-1% of gradients suffices.
• Error Feedback: The error feedback mechanism ensures convergence despite extreme sparsification.

Gradient Sparsification is sending only the important gradients — reducing communication by orders of magnitude while maintaining training quality.