Knowledge Distillation
Knowledge Distillation — a model compression technique where a small "student" network learns to mimic the behavior of a large "teacher" network, achieving near-teacher accuracy at a fraction of the size.
How It Works
1. Train a large, accurate teacher model
2. Run teacher on training data → collect "soft labels" (probability distributions, not just the predicted class)
3. Train student to match both:
- Hard labels (ground truth)
- Soft labels from teacher (with temperature scaling)
Why Soft Labels?
- Hard label: [0, 0, 1, 0] — "this is a cat"
- Soft label: [0.01, 0.05, 0.90, 0.04] — "this is mostly cat, slightly dog-like"
- Soft labels encode "dark knowledge" — relationships between classes that hard labels miss
Temperature Scaling
$$p_i = \frac{\exp(z_i / T)}{\sum \exp(z_j / T)}$$
- $T > 1$: Softens the distribution (reveals more structure)
- Typical: $T = 3$–$20$ during distillation
Results
- Student (1/10th the size) often achieves 95-99% of teacher accuracy
- DistilBERT: 60% smaller, 60% faster, retains 97% of BERT's performance
- Used in deploying LLMs to mobile/edge devices
Distillation is one of the most practical compression techniques — it's how large AI models get deployed to real-world applications.