Weight Quantization LLM

Weight Quantization for LLMs is the model compression technique that reduces the numerical precision of neural network weights from 16-bit floating point to 4-bit or 8-bit integers — shrinking model size by 2-4x and proportionally reducing memory bandwidth requirements during inference, enabling large language models that would require multiple GPUs to run on a single consumer GPU with minimal quality degradation.

Why Quantization Is Critical for LLM Deployment

A 70B-parameter model in FP16 requires 140 GB of memory — exceeding any single consumer GPU. Quantizing to 4-bit reduces this to ~35 GB, fitting on a single 48GB GPU (RTX 4090 or A6000). Since LLM inference is memory-bandwidth-bound (the bottleneck is reading weights from memory, not computing), 4x smaller weights → up to 4x faster token generation.

Quantization Approaches

• Round-to-Nearest (RTN): Simply round each FP16 weight to the nearest INT4/INT8 value using a per-channel or per-group scale factor. Fast but produces significant accuracy loss at 4-bit, especially for models with outlier weights.

• GPTQ (Frantar et al., 2022): An optimal per-column quantization method based on the Optimal Brain Quantization framework. For each weight column, GPTQ finds the best INT4 values by minimizing the quantization error on a calibration dataset, adjusting remaining unquantized weights to compensate for the error already introduced. Processes one column at a time in a single pass. Result: 4-bit quantization with negligible perplexity increase for 7B-70B models.

• AWQ (Activation-Aware Weight Quantization): Observes that a small fraction (~1%) of weights are disproportionately important because they correspond to large activations. AWQ protects these salient weights by applying per-channel scaling that reduces their quantization error at the expense of less-important weights. Simpler than GPTQ, comparable quality, and faster calibration.

• GGUF / llama.cpp Quantization: Practical quantization formats optimized for CPU inference. Supports multiple quantization levels (Q4_K_M, Q5_K_M, Q8_0) with per-block scale factors and optional importance-weighted mixed precision. The dominant format for local LLM inference.

• SqueezeLLM / QuIP#: Research methods achieving near-lossless 2-3 bit quantization using incoherence processing (rotating weights to spread information uniformly) and lattice codebooks (multi-dimensional quantization that better preserves weight relationships).

Mixed-Precision Quantization

Not all layers are equally sensitive to quantization. Attention QKV projections and the first/last layers are typically more sensitive. Mixed-precision approaches assign higher precision (8-bit) to sensitive layers and lower precision (4-bit) to robust layers, optimizing the quality-size tradeoff.

Quality Impact

Weight Quantization is the compression technology that democratized LLM access — making models that require data-center GPUs at full precision runnable on consumer hardware by exploiting the fact that neural network weights contain far more numerical precision than they actually need.