Top-K Gradient Sparsification

Top-K Gradient Sparsification is the most common gradient sparsification strategy — selecting only the K gradient components with the largest magnitude for communication, where K is typically 0.1-1% of the total gradient dimension.

Top-K Algorithm

• Compute: Compute the full gradient locally.
• Select: Find the top-K components by absolute magnitude.
• Communicate: Send only these K (index, value) pairs — ~99% compression.
• Error Feedback: Accumulate the unsent components and add to the next gradient: $e_{t+1} = g_t - TopK(g_t + e_t)$.

Why It Matters

• Convergence Guarantee: With error feedback, top-K sparsification converges to the same solution as full gradient communication.
• All-Reduce: Sparse all-reduce is more complex than dense all-reduce — specialized communication primitives needed.
• Hardware: Modern accelerators (GPUs, TPUs) have high compute-to-communication ratios — sparsification exploits this.

Top-K Sparsification is selecting the most impactful gradients — sending only the largest updates for massive communication savings.