LoRA
LoRA (Low-Rank Adaptation) is the parameter-efficient fine-tuning method that injects small trainable low-rank matrices into frozen pretrained model layers — enabling fine-tuning of billion-parameter models on consumer GPUs by training only 0.1-1% of total parameters while achieving 90-100% of full fine-tuning quality, democratizing LLM customization.
Core Idea
- Original weight matrix: W₀ ∈ R^(d×d) (frozen, not updated).
- LoRA adds: ΔW = B × A where A ∈ R^(r×d), B ∈ R^(d×r), rank r << d.
- Forward pass: $h = (W_0 + \frac{\alpha}{r} BA)x$.
- Only A and B are trained — W₀ stays frozen.
Why It Works
- Aghajanyan et al. (2021): Pretrained models have low intrinsic dimensionality.
- Fine-tuning changes are concentrated in a low-rank subspace.
- Rank r = 8-64 captures most of the adaptation signal (d = 4096 for a 7B model).
Parameter Efficiency
| Model | Full FT Params | LoRA (r=16) | Reduction |
|---|---|---|---|
| LLaMA-7B | 6.7B | ~4M | 1675x |
| LLaMA-13B | 13B | ~6.5M | 2000x |
| LLaMA-70B | 70B | ~33M | 2121x |
Memory Savings
- Full fine-tuning 7B model: ~120GB (weights + gradients + optimizer states in fp32).
- LoRA fine-tuning 7B model: ~16-24GB (frozen weights in bf16 + small trainable params).
- Fits on a single 24GB GPU (RTX 4090) — vs. 4+ A100s for full fine-tuning.
QLoRA (Quantized LoRA)
- Quantize frozen base model to 4-bit (NF4 quantization).
- LoRA adapters remain in bf16/fp16.
- Backprop through quantized weights using double quantization.
- Result: Fine-tune 65B model on a single 48GB GPU (A6000).
- Quality: Within 1% of full 16-bit fine-tuning on most benchmarks.
Practical Configuration
| Parameter | Typical Value | Notes |
|---|---|---|
| Rank (r) | 8-64 | Higher = more capacity, more params |
| Alpha (α) | 16-32 | Scaling factor, often set to 2×rank |
| Target modules | q_proj, v_proj (attention) | Can also target k_proj, o_proj, FFN |
| Dropout | 0.05-0.1 | On LoRA layers |
| Learning rate | 1e-4 to 3e-4 | Higher than full fine-tuning |
LoRA Variants
- DoRA: Decompose weight into magnitude and direction, LoRA adapts direction.
- AdaLoRA: Adaptive rank allocation — more rank for important layers.
- LoRA+: Different learning rates for A and B matrices.
- Tied LoRA: Share LoRA weights across layers.
Merging and Serving
- After training: Merge LoRA weights into base model: $W_{merged} = W_0 + \frac{\alpha}{r}BA$.
- Merged model has zero inference overhead — identical architecture to base.
- Multiple LoRA adapters can be swapped at inference time for different tasks.
LoRA is the technique that made LLM fine-tuning accessible to everyone — by reducing the hardware requirements from a cluster of A100s to a single consumer GPU, it enabled the explosion of open-source fine-tuned models and custom AI applications.