Quantization
Quantization stores and computes a neural network in fewer bits — mapping high-precision FP32 weights and activations onto a small set of low-precision levels (INT8, INT4, FP8) so the model takes less memory and runs faster.
It is a mapping from a continuous range onto a discrete grid. A quantizer picks a scale s and a zero-point z, then represents each real value x as the integer q = round(x / s) + z and recovers an approximation x' = s(q - z). INT8 gives 256 levels, INT4 only 16. The art is choosing s — and whether to use one scale per tensor, per channel, or per block — so the grid lands where the weights actually cluster.
Why it pays off. Cutting FP32 to INT8 shrinks the model 4x and cuts memory traffic 4x, which — per the roofline — is exactly what helps a memory-bound LLM. The multiplies also become cheap integer or low-precision operations that tensor cores run several times faster. The cost is rounding error: too few levels, or a few outlier values stretching the scale, and accuracy drops.
| Approach | When quantized | Accuracy | Cost | Typical use |
|---|---|---|---|---|
| PTQ (post-training) | after training | good with care | minutes | most INT8 deploys |
| QAT (quant-aware) | during training | best | full retrain | aggressive low-bit |
| Weight-only | weights only, acts FP16 | high | simple | LLM inference (GPTQ, AWQ) |
| Dynamic | activations at runtime | good | slight overhead | activation-sensitive nets |
Bit-width is a spectrum, and outliers set the floor. INT8 is routine; INT4 and FP4 need care because a handful of outlier activations dominate the range and inflate the rounding error for everything else. Per-channel scales, group-wise quantization, and outlier-aware methods (SmoothQuant, AWQ, GPTQ) exist precisely to keep the grid useful as the bit-width shrinks.
Read quantization through a quant lens rather than an accuracy-table lens: it is a knob on arithmetic intensity and model footprint, not merely a compression trick. Every bit removed multiplies effective bandwidth and tensor-core throughput, so the real question is how far levels can drop before rounding error crosses the task's tolerance — a measured bytes-versus-error trade, not an assumed one.