Exl2

EXL2 is an advanced quantization format for ExLlamaV2 that uses dynamic per-layer bit allocation to achieve optimal quality-size trade-offs for GPU inference of large language models. Key innovation: adaptively assigns different quantization bits to each layer based on sensitivity—important layers get more bits (4-8), less critical layers get fewer (2-4)—vs. uniform quantization. Bit allocation: typically averages 3-5 bits per weight overall while preserving quality better than fixed-bit approaches. ExLlamaV2: CUDA-optimized inference engine for quantized LLaMA-style models, achieving very fast generation speeds. Performance: 50-100+ tokens/second on consumer GPUs (RTX 3090/4090) for 7B-70B models with EXL2. Compression: 70B model in <20GB VRAM achievable with aggressive quantization, enabling local inference. Calibration: requires calibration dataset to determine optimal bit allocation per layer. Quality retention: at equivalent average bits, EXL2 typically outperforms GPTQ and AWQ due to adaptive allocation. Integration: used via ExLlamaV2 Python library or front-ends like Text Generation WebUI. Comparison: GPTQ (uniform bits, widely supported), AWQ (activation-aware, fast), EXL2 (adaptive bits, potentially best quality/size). Model availability: quantized versions available on Hugging Face in EXL2 format. Leading quantization format for local LLM inference balancing quality and memory efficiency.