Ternary Gradients

Ternary Gradients is a gradient quantization scheme that compresses each gradient component to one of three values: {-1, 0, +1} — achieving very high compression while preserving sparsity, as zero gradients are explicitly represented.

Ternary Quantization Methods

• TernGrad: Stochastic ternary quantization — $\hat{g}_i \in {-s, 0, +s}$ where $s$ is a scaling factor.
• Threshold-Based: Components with magnitude below a threshold are set to 0, others to $pm s$.
• Stochastic Rounding: $P(\hat{g}_i = s \cdot \text{sign}(g_i)) = |g_i|/s$ — unbiased with controlled variance.
• Encoding: {-1, 0, +1} requires ~1.585 bits per component — encode efficiently with run-length encoding.

Why It Matters

• Sparsity Aware: Unlike 1-bit SGD, ternary gradients preserve gradient sparsity — zero gradients stay zero.
• Unbiased: Stochastic ternary quantization is an unbiased estimator — convergence is theoretically guaranteed.
• Hardware Friendly: Ternary operations can be implemented efficiently on specialized hardware.

Ternary Gradients are the three-symbol gradient alphabet — compressing gradients to {-1, 0, +1} for efficient communication with sparsity awareness.