Knowledge Distillation
Knowledge Distillation is the model compression technique where a smaller "student" network is trained to mimic the output behavior of a larger, more accurate "teacher" network — transferring the teacher's learned knowledge through soft probability distributions rather than hard labels, enabling deployment of compact models that retain 90-99% of the teacher's accuracy at a fraction of the size and computation.
Core Idea (Hinton et al., 2015)
- Teacher output (softmax with temperature T): $p_i^T = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)}$.
- At high temperature (T=4-20): Softmax outputs reveal inter-class relationships (e.g., "3" looks more like "8" than like "7").
- These soft labels carry richer information than one-hot hard labels.
- Student learns to match teacher's soft distribution → learns the teacher's reasoning patterns.
Distillation Loss
$L = \alpha \cdot T^2 \cdot KL(p^T_{teacher} p^T_{student}) + (1-\alpha) \cdot CE(y, p_{student})$
- First term: Match teacher's soft predictions (KL divergence).
- Second term: Match ground truth labels (cross-entropy).
- α: Balance between teacher guidance and ground truth (typically 0.5-0.9).
- T²: Compensates for gradient magnitude changes at high temperature.
Types of Distillation
| Type | What's Transferred | Example |
|---|---|---|
| Response-based | Final layer outputs (logits) | Classic Hinton distillation |
| Feature-based | Intermediate layer activations | FitNets, attention transfer |
| Relation-based | Relationships between samples | Relational KD, CRD |
| Self-distillation | Same architecture, deeper→shallower | Born-Again Networks |
| Online distillation | Multiple models teach each other | Deep Mutual Learning |
LLM Distillation
- Alpaca/Vicuna approach: Generate training data from GPT-4 → fine-tune smaller model.
- Not classic distillation (no soft labels) — actually data distillation or imitation learning.
- Logit distillation: Access to teacher logits for each token → train student to match distribution.
- DistilBERT: 40% smaller, 60% faster, retains 97% of BERT performance.
- TinyLlama: 1.1B model trained on same data as larger models — competitive performance.
Practical Guidelines
- Teacher-student size gap: Student should be 2-10x smaller. Too large a gap reduces distillation effectiveness.
- Temperature: Start with T=4, tune in range [2, 20].
- Feature distillation: Add projection layers if teacher/student feature dimensions differ.
- Ensemble teachers: Distilling from an ensemble of teachers gives better results than a single teacher.
Knowledge distillation is the primary technique for deploying large models in resource-constrained environments — from compressing BERT for mobile deployment to creating smaller LLMs from GPT-class teachers, distillation bridges the gap between research-scale accuracy and production-scale efficiency.