Per-Tensor Quantization
Per-tensor quantization uses a single set of quantization parameters (scale and zero-point) for an entire tensor, regardless of its shape or the variance across its dimensions. This is the simplest and most common quantization granularity.
How It Works
For a tensor $T$ with arbitrary shape:
Where:
- $s$ is the scale factor (computed from the tensor's min/max values).
- $z$ is the zero-point offset (for asymmetric quantization).
Scale Calculation
For 8-bit quantization:
(For symmetric quantization, use $max(|T|)$ instead.)
Advantages
- Simplicity: One scale and zero-point for the entire tensor — minimal storage overhead.
- Fast Inference: Dequantization is straightforward with no per-channel or per-element overhead.
- Hardware Friendly: Most quantization-aware hardware accelerators (TPUs, NPUs) are optimized for per-tensor quantization.
Disadvantages
- Suboptimal for Heterogeneous Data: If different regions of the tensor have very different value ranges, per-tensor quantization wastes precision. For example, if one channel has values in [-0.1, 0.1] and another in [-10, 10], the shared scale must accommodate [-10, 10], losing precision for the first channel.
- Outliers: A single outlier value can dominate the scale calculation, reducing precision for the majority of values.
When to Use Per-Tensor
- Activations: Standard choice for activation quantization because per-channel activations would require runtime overhead.
- Small Tensors: For tensors with relatively uniform value distributions.
- Hardware Constraints: When deploying to hardware that only supports per-tensor quantization.
Comparison to Per-Channel
| Aspect | Per-Tensor | Per-Channel |
|---|---|---|
| Parameters | 1 scale + 1 zero-point | N scales + N zero-points (N = channels) |
| Accuracy | Lower (for heterogeneous data) | Higher |
| Speed | Fastest | Slightly slower |
| Storage | Minimal | Small overhead |
| Use Case | Activations, uniform data | Weights, heterogeneous data |
Example
For a weight tensor with shape [64, 128, 3, 3] (64 output channels):
- Per-Tensor: Compute $min$ and $max$ across all 73,728 values, derive one scale.
- Per-Channel: Compute $min$ and $max$ for each of the 64 output channels separately, derive 64 scales.
Per-tensor quantization is the default choice for activations and a reasonable baseline for weights, though per-channel quantization typically provides better accuracy for weights.