Error Feedback in Compressed Communication

Error Feedback (Memory) is a mechanism that compensates for gradient compression losses by accumulating unsent gradient components locally — the accumulated error is added to the next round's gradient before compression, ensuring that all gradient information is eventually communicated.

How Error Feedback Works

• Compress: Apply compression $C(g_t + e_t)$ to the gradient plus accumulated error.
• Communicate: Send the compressed gradient $C(g_t + e_t)$.
• Accumulate: Store the compression error: $e_{t+1} = (g_t + e_t) - C(g_t + e_t)$.
• Next Round: Add accumulated error to next gradient: $g_{t+1} + e_{t+1}$.

Why It Matters

• Convergence Fix: Without error feedback, aggressive compression prevents convergence. With error feedback, convergence is guaranteed.
• No Information Loss: Every gradient component is eventually communicated — just delayed, not lost.
• Universal: Error feedback works with any compression method (top-K, random, quantization).

Error Feedback is remembering what you didn't send — accumulating compression residuals to ensure no gradient information is permanently lost.