Model Quantization Inference
Model Quantization is the inference optimization technique that reduces the numerical precision of neural network weights and activations from 32-bit or 16-bit floating-point to lower bit-widths (8-bit, 4-bit, or even 2-bit integers) — shrinking model memory footprint by 2-8x, accelerating computation on hardware with integer execution units, and enabling deployment of large models on resource-constrained devices with minimal quality degradation.
Why Quantize
A 70B parameter model in FP16 requires 140 GB of memory — exceeding the capacity of any single consumer GPU. Quantizing to 4-bit reduces this to ~35 GB, fitting on a single 48GB GPU. Beyond memory, integer arithmetic is 2-4x faster than floating-point on most hardware, and reduced memory bandwidth (the primary bottleneck for LLM inference) directly increases tokens-per-second.
Post-Training Quantization (PTQ)
Quantize a pre-trained model without retraining:
- Round-to-Nearest (RTN): Simply round each weight to the nearest quantized value. Works well at INT8; significant quality loss at INT4.
- GPTQ: Uses approximate second-order information (Hessian) to quantize weights one at a time, adjusting remaining weights to compensate for the quantization error. Achieves near-lossless INT4 weight quantization for LLMs.
- AWQ (Activation-Aware Weight Quantization): Identifies the small fraction (~1%) of weight channels that are critical for maintaining accuracy (those corresponding to large activation magnitudes) and protects them with per-channel scaling before quantization.
- SqueezeLLM / QuIP: Use non-uniform quantization and incoherence processing to push quality at extreme (2-3 bit) compression.
Quantization-Aware Training (QAT)
Simulate quantization during training by inserting fake-quantization nodes that round weights/activations during the forward pass but pass gradients through using the straight-through estimator. The model learns to be robust to quantization noise, consistently outperforming PTQ at the same bit-width but requiring a full training run.
Quantization Formats
| Format | Bits | Memory Ratio | Quality Impact | Use Case |
|---|---|---|---|---|
| FP16/BF16 | 16 | 1x (baseline) | None | Training, high-quality inference |
| INT8 (W8A8) | 8 | 0.5x | Negligible | Production serving |
| INT4 (W4A16) | 4 weights, 16 activations | 0.25x weights | Small (<1% accuracy) | Consumer GPU deployment |
| GGUF Q4_K_M | 4-6 mixed | ~0.3x | Small | CPU/edge inference (llama.cpp) |
| INT2-3 | 2-3 | 0.12-0.19x | Moderate | Research/extreme compression |
Mixed-Precision and Group Quantization
Rather than quantizing all weights to the same precision, modern methods use group quantization (quantize in blocks of 32-128 weights with per-group scale factors) and mixed precision (keep sensitive layers at higher precision). This provides fine-grained control over the accuracy-compression tradeoff.
Model Quantization is the compression technique that made billion-parameter AI accessible on consumer hardware — proving that neural networks are massively over-precise and that most of their intelligence survives dramatic precision reduction.