Powersgd

PowerSGD is a low-rank gradient compression method that approximates gradient matrices with their top-$k$ singular vectors — using power iteration to efficiently compute a low-rank approximation, achieving high compression with better accuracy than sparsification or quantization.

How PowerSGD Works

• Low-Rank: Approximate gradient matrix $G \approx P Q^T$ where $P$ and $Q$ are tall, thin matrices (rank $k$).
• Power Iteration: Use 1-2 steps of power iteration starting from the previous $Q$ to quickly approximate top singular vectors.
• Communication: Communicate $P$ and $Q$ (total size = $k(m+n)$) instead of $G$ (size = $m \times n$) — compression ratio = $mn / k(m+n)$.
• Error Feedback: Accumulate the compression residual for next iteration.

Why It Matters

• Better Trade-Off: PowerSGD achieves better accuracy-compression trade-offs than sparsification or quantization.
• Warm Start: Reusing the previous iteration's $Q$ makes power iteration converge in just 1-2 steps.
• Practical: Integrated into PyTorch's distributed data parallel (DDP) as a built-in communication hook.

PowerSGD is low-rank gradient communication — transmitting compact matrix factorizations instead of full gradients for efficient, high-quality compression.