Accordion

Accordion is an adaptive gradient compression framework that dynamically adjusts the compression ratio during training — using more compression when the model is making rapid progress (gradient information is less critical) and less compression during delicate convergence phases.

How Accordion Works

• Monitoring: Track a training metric (gradient variance, loss change, learning rate) to assess the training phase.
• Adaptive Ratio: High compression when gradients are informative (early training), low compression near convergence.
• Scheduler: Compression ratio follows a schedule synchronized with the learning rate schedule.
• Any Compressor: Works with any base compressor (top-K, random-K, PowerSGD, quantization).

Why It Matters

• Optimal Efficiency: Different training phases have different communication sensitivity — Accordion exploits this.
• No Accuracy Loss: By being conservative when it matters and aggressive when it doesn't, Accordion achieves lossless training.
• Automatic: No manual tuning of compression ratios — the framework adapts automatically.

Accordion is breathing with the training — dynamically adjusting communication compression to match each training phase's sensitivity to gradient accuracy.