Knowledge Distillation Model Compression

Knowledge Distillation is the model compression technique where a smaller "student" network is trained to replicate the behavior of a larger, more accurate "teacher" network — learning from the teacher's soft probability outputs (which encode inter-class relationships) rather than hard ground-truth labels, achieving 90-99% of teacher accuracy at a fraction of the computational cost.

Distillation Framework:
- Teacher Model: large, high-accuracy model that has been fully trained — may be an ensemble of models for even richer soft labels; teacher is frozen (not updated) during distillation
- Student Model: compact model architecture designed for deployment — typically 3-10× fewer parameters than teacher; architecture can differ from teacher (e.g., teacher is ResNet-152, student is MobileNet)
- Temperature Scaling: softmax outputs computed with temperature T — higher T (typically 2-20) produces softer probability distributions that reveal more information about inter-class similarities; T=1 recovers standard softmax
- Distillation Loss: KL divergence between teacher and student soft distributions scaled by T² — combined with standard cross-entropy loss on hard labels; α parameter controls the weighting (typically α=0.5-0.9 for distillation loss)

Distillation Variants:
- Response-Based: student matches teacher's final output logits — simplest form; captures the teacher's class relationship knowledge encoded in soft probabilities
- Feature-Based: student matches intermediate feature representations of the teacher — FitNets, Attention Transfer, and PKT methods align hidden layer activations, transferring structural knowledge about feature hierarchies
- Relation-Based: student preserves the relational structure between samples as encoded by the teacher — Relational Knowledge Distillation (RKD) preserves pairwise distance and angle relationships in embedding space
- Self-Distillation: model distills knowledge from its own deeper layers to shallower layers, or from a trained version of itself — Born-Again Networks show iterative self-distillation can progressively improve student beyond teacher accuracy

Advanced Techniques:
- Online Distillation: teacher and student train simultaneously, mutually learning from each other — Deep Mutual Learning shows peer networks can teach each other without a pre-trained teacher
- Data-Free Distillation: generates synthetic training data using the teacher's batch normalization statistics or a trained generator — useful when original training data is unavailable due to privacy or storage constraints
- Task-Specific Distillation: DistilBERT reduces BERT parameters by 40% while retaining 97% performance — uses triple loss: masked language model, distillation, and cosine embedding loss
- Multi-Teacher Distillation: student learns from multiple teachers specializing in different domains or architectures — teacher contributions can be equally weighted or dynamically adjusted based on per-sample confidence

Knowledge distillation is the cornerstone of efficient model deployment — enabling state-of-the-art accuracy on resource-constrained devices (mobile phones, edge processors, embedded systems) by transferring the "dark knowledge" encoded in large models into compact, fast inference networks.