LoRA

LoRA (Low-Rank Adaptation) is the parameter-efficient fine-tuning technique that injects trainable low-rank decomposition matrices into frozen pretrained model weights — enabling fine-tuning of large language models with 10,000× fewer trainable parameters than full fine-tuning, by approximating weight updates as a product of two small matrices (W = W₀ + BA where B ∈ R^(d×r), A ∈ R^(r×k), rank r ≪ min(d,k)), making it practical to adapt billion-parameter models on consumer GPUs.

Core Idea: Low-Rank Weight Updates

• Full fine-tuning: Update all W₀ ∈ R^(d×k) — too expensive for LLMs.
• LoRA insight: Weight updates during fine-tuning have low intrinsic rank — the update ΔW ≈ BA where r = 4–64 captures most useful adaptation.
• Merged at inference: W = W₀ + BA → no extra latency (matrices merged before deployment).
• Trainable params: r×(d+k) vs d×k. For d=k=4096, r=8: 65K vs 16M parameters.

LoRA Architecture

import torch, torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, in_features, out_features, rank=8, alpha=16):
        super().__init__()
        self.W0 = nn.Linear(in_features, out_features, bias=False)  # frozen
        self.A = nn.Linear(in_features, rank, bias=False)           # trainable
        self.B = nn.Linear(rank, out_features, bias=False)          # trainable
        self.scale = alpha / rank    # scaling factor
        # Initialize: A ~ N(0,1), B = 0 (so LoRA starts at zero update)
        nn.init.kaiming_uniform_(self.A.weight)
        nn.init.zeros_(self.B.weight)
        self.W0.weight.requires_grad = False  # freeze base weights

    def forward(self, x):
        return self.W0(x) + self.scale * self.B(self.A(x))

Where to Apply LoRA

QLoRA: Quantized LoRA

• QLoRA (Dettmers et al., 2023): Load pretrained model in 4-bit NF4 quantization → add LoRA adapters in bfloat16.
• NF4 (Normal Float 4-bit): Quantization levels chosen for normally distributed weights → minimal quantization error.
• Paged optimizers: Offload optimizer states to CPU RAM when GPU OOM → enables 65B model fine-tuning on single 48GB GPU.
• Typical result: QLoRA matches full 16-bit fine-tuning quality at ~30% GPU memory.

Practical LoRA Settings

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,                    # rank
    lora_alpha=32,           # scaling (alpha/r = 2.0 is common)
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,       # regularization
    bias="none",             # don't train bias terms
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()  # Shows << 1% trainable

PEFT Method Comparison

LoRA Variants

• DoRA (Weight-Decomposed LoRA): Decomposes weight into magnitude + direction; adapts direction via LoRA → better initialization.
• LoRA+: Different learning rates for A and B matrices → faster convergence.
• AdaLoRA: Adaptive rank allocation — important layers get higher rank, prunes unimportant singular values.
• LoftQ: Quantization-aware LoRA initialization — reduces gap between NF4 quantization and full precision.

LoRA and PEFT are the enabling technology for democratizing large language model fine-tuning — by reducing trainable parameters from billions to millions while preserving 95%+ of full fine-tuning quality, LoRA makes domain-specific LLM adaptation accessible on consumer hardware, turning what was a month-long distributed training job into an overnight single-GPU experiment and spawning the entire open-source fine-tuned LLM ecosystem.