Post Training Quantization

Post-Training Quantization (PTQ) is the model compression technique that reduces the numerical precision of neural network weights and activations after training is complete — without requiring retraining or fine-tuning, converting float32/bfloat16 models to int8, int4, or lower precision to reduce memory footprint by 2–8× and increase inference throughput by 1.5–4× on hardware with quantized compute support, at a small accuracy cost that modern algorithms minimize through careful calibration.

Why LLMs Need Specialized PTQ

• Standard PTQ (per-tensor, per-channel) works well for CNNs but struggles with LLMs.
• LLM activations contain outliers: a few channels have 100× larger values than others.
• Naively quantizing these outliers causes massive accuracy loss.
• Solution: per-channel/group quantization, outlier-aware methods, weight-only quantization.

GPTQ (Frantar et al., 2022)

• Applies Optimal Brain Quantization (OBQ) row-by-row to transformer weight matrices.
• Quantizes weights to int4 using second-order Hessian information → minimizes quantization error.
• Key insight: Quantize one weight at a time, update remaining weights to compensate for error.
• Speed: Quantizes 175B GPT model in ~4 hours on a single GPU.
• Result: int4 GPTQ quality ≈ int8 naive quantization for most LLMs.

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

quantize_config = BaseQuantizeConfig(
    bits=4,           # int4
    group_size=128,   # quantize in groups of 128 weights
    desc_act=False,   # disable activation order for speed
)
model = AutoGPTQForCausalLM.from_pretrained(model_path, quantize_config)
model.quantize(calibration_data)  # Calibrate on ~128 samples

AWQ (Activation-aware Weight Quantization)

• Observes that a small fraction (~1%) of weights are "salient" — high activation scale → large quantization error if rounded.
• Solution: Scale salient weights up before quantization → scale activations down to compensate.
• Math: (s·W)·(X/s) = W·X but (s·W) quantizes more accurately since s > 1.
• No retraining: Only ~1% of weights are scaled, rest are straightforward int4.
• Result: AWQ generally outperforms GPTQ at very low bit-widths (< 4 bit).

SmoothQuant

• Problem: Activation outliers make int8 activation quantization difficult.
• Solution: Transfer quantization difficulty from activations to weights via per-channel scaling.
• Math: Y = (Xdiag(s)⁻¹)·(diag(s)W) where s smooths activation dynamic range.
• Enables W8A8 (int8 weights + int8 activations) → uses tensor core INT8 arithmetic → 1.6–2× faster than FP16.

Quantization Granularity

Key Metrics and Trade-offs

• Perplexity delta: int4 GPTQ: +0.2–0.5 perplexity on WikiText2 vs FP16 baseline.
• Memory reduction: FP16 (2 bytes) → INT4 (0.5 bytes) = 4× reduction.
• Throughput: INT4 weight-only: 1.5–2.5× faster generation (memory bandwidth limited).
• W8A8: 1.5–2× faster for batch inference (compute-limited scenarios).

Calibration Data

• PTQ requires small calibration dataset (128–512 samples) to compute activation statistics.
• Quality matters: calibration data should match downstream task distribution.
• Common: WikiText, C4, or task-specific examples.

Post-training quantization is the practical gateway to deploying state-of-the-art LLMs on accessible hardware — by compressing 70B parameter models from 140GB in FP16 to 35GB in INT4 without costly retraining, PTQ methods like GPTQ and AWQ have made it possible to run frontier-scale models on single workstation GPUs, democratizing LLM inference and enabling the local AI ecosystem that powers privacy-preserving, offline-capable AI applications.