Gptq
GPTQ (Generative Pre-trained Transformer Quantization) is a post-training quantization method that achieves 3-4 bit weight quantization for large language models with minimal accuracy loss by using second-order information and layer-wise quantization with calibration data. Method: (1) layer-wise quantization (quantize one layer at a time, keeping others in FP16), (2) optimal brain quantization (OBQ—use Hessian inverse to determine quantization order and compensate for errors), (3) calibration data (128-1024 samples—compute activations and Hessian). Key innovation: compensate for quantization error by adjusting remaining unquantized weights—when quantizing weight w_i, adjust other weights to minimize output error using Hessian information. Algorithm: (1) compute Hessian H = ∂²L/∂W² for layer weights (approximate from calibration data), (2) for each weight in order: quantize weight, compute error, adjust remaining weights using H⁻¹ to compensate. Quantization targets: (1) 4-bit (most common—3.5× memory reduction, good accuracy), (2) 3-bit (aggressive—5× reduction, some accuracy loss), (3) 2-bit (extreme—8× reduction, significant degradation). Group quantization: quantize weights in groups (e.g., 128 weights per group)—separate scale per group improves accuracy vs. per-channel. Performance: 4-bit GPTQ models achieve <1% perplexity increase on LLaMA, Mistral, and other LLMs—enables running 70B models on consumer GPUs (24GB VRAM). Inference: (1) dequantize weights on-the-fly during computation, (2) use INT4 matrix multiplication (CUDA kernels), (3) 2-3× speedup vs. FP16 on memory-bound workloads. Comparison: (1) GPTQ (post-training, uses calibration data, high accuracy), (2) AWQ (activation-aware, protects important weights), (3) GGML/GGUF (CPU-focused, various bit widths), (4) bitsandbytes (simpler, slightly lower accuracy). Tools: AutoGPTQ (Python library), ExLlama (fast inference), transformers (Hugging Face integration). Limitations: (1) requires calibration data (representative of target distribution), (2) quantization time (hours for 70B models), (3) some accuracy loss (task-dependent). GPTQ has become standard for deploying large language models on consumer hardware, democratizing access to powerful models.