Logarithmic Quantization
Logarithmic quantization applies quantization on a logarithmic scale rather than a linear scale, allocating more precision to smaller values and less precision to larger values. This approach is particularly effective for neural network weights and activations that follow exponential or power-law distributions.
How It Works
- Linear Quantization: Divides the value range into equal intervals. A value of 0.1 and 0.2 get the same precision as 10.0 and 10.1.
- Logarithmic Quantization: Divides the logarithmic space into equal intervals. Smaller values (near zero) receive finer granularity, while larger values are coarsely quantized.
Mathematical Representation
For a value $x$, logarithmic quantization computes:
$$q = \text{round}(\log_2(|x|) \cdot s) \cdot \text{sign}(x)$$
Where $s$ is a scale factor. Dequantization reconstructs:
$$\hat{x} = 2^{q/s} \cdot \text{sign}(x)$$
Advantages
- Better Dynamic Range: Captures both very small and very large values effectively without wasting quantization levels.
- Natural Fit for Weights: Neural network weights often follow distributions where most values are small, making logarithmic quantization more efficient than linear.
- Reduced Quantization Error: For exponentially distributed data, logarithmic quantization minimizes mean squared error compared to linear quantization.
Applications
- Model Compression: Quantize weights in deep networks where weight magnitudes span several orders of magnitude.
- Audio Processing: Audio signals have logarithmic perceptual characteristics (decibels), making log quantization natural.
- Gradient Compression: Gradients in distributed training often have exponential distributions.
Comparison to Linear Quantization
| Aspect | Linear | Logarithmic |
|---|---|---|
| Precision Distribution | Uniform across range | Higher for small values |
| Dynamic Range | Limited | Excellent |
| Implementation | Simple | Slightly more complex |
| Best For | Uniform distributions | Exponential distributions |
Logarithmic quantization is less common than linear quantization but provides significant advantages for specific data distributions, particularly in model compression and audio applications.