Model Compression

Model compression reduces model size and compute requirements through techniques like pruning, quantization, and distillation. Why compress: Deployment on edge devices, reduce serving costs, lower latency, fit in memory constraints. Main techniques: Quantization: Reduce precision (FP32 to INT8, INT4). 2-4x size reduction. Pruning: Remove unimportant weights or structures. Variable reduction. Distillation: Train small model to mimic large one. Design smaller architecture. Combined approaches: Often stack techniques - distill, then quantize and prune. Accuracy trade-off: Compression usually reduces accuracy slightly. Goal is minimal degradation for significant efficiency gains. Structured vs unstructured: Structured compression (remove whole channels/layers) gives real speedup. Unstructured (sparse weights) needs specialized hardware. Tools: TensorRT (NVIDIA), OpenVINO (Intel), ONNX Runtime, Core ML, llama.cpp, GPTQ, AWQ. LLM compression: Quantization most impactful (4-bit models common). Pruning and distillation also used. Evaluation: Measure accuracy retention, actual speedup, memory reduction. Paper claims vs real deployment may differ.