Knowledge Distillation Teacher Student

Knowledge Distillation is the training paradigm where smaller student networks learn from larger teacher model via soft target distributions — achieving substantial parameter reduction (40%+ compression) with minimal performance loss, enabling deployment on edge/mobile devices.

Core Distillation Framework:
- Teacher-student training: large pretrained teacher model guides smaller student network; knowledge transfer via probability distributions
- Soft targets: teacher output softmax logits (before argmax) preserve uncertainty and inter-class relationships compared to hard labels
- Distillation loss: combines student cross-entropy on original labels with KL divergence from teacher soft targets
- Temperature scaling: softening both teacher and student outputs via Softmax(z/T); higher T (4-10) creates gentler probability landscape for learning

Hinton's Distillation Objective:
- KL divergence loss: measures divergence between student and teacher probability distributions; guides student learning beyond hard labels
- Weighted combination: total loss = α·cross_entropy(student, hard_labels) + (1-α)·KL_divergence(student, teacher_soft_targets)
- Temperature effect: Softmax(z/T) for T > 1 reduces peaks in distributions; smoother gradients aid student learning
- Optimal temperature: typically 3-20 depending on dataset; higher T for more complex task knowledge transfer

Knowledge Distillation Variants:
- Response-based distillation: student matches final output distribution; most common; effective for classification tasks
- Feature-based distillation: intermediate layer feature matching between student/teacher; additional guidance beyond final output
- Relation-based distillation: student learns relationships between data examples from teacher; meta-knowledge transfer

DistilBERT and TinyBERT:
- DistilBERT: 40% parameter reduction from BERT (66M vs 110M), 60% speedup, 97% GLUE performance retention
- Distillation + pruning + quantization: combined compression techniques achieve 2-4x speedup with minimal quality loss
- TinyBERT: task-specific distillation for further compression; knowledge distillation at intermediate layers
- Applications: mobile inference, edge deployment, real-time inference with resource constraints

Deployment Benefits:
- Smaller model size: enables deployment on smartphones, IoT devices, browser-based ML
- Inference latency reduction: fewer parameters, smaller memory footprint → faster inference
- Energy efficiency: reduced computation and memory bandwidth crucial for battery-powered devices
- Cost reduction: fewer parameters → cheaper inference infrastructure

Knowledge distillation successfully transfers learned representations from large teacher models to compact students — enabling efficient deployment without significant performance degradation across NLP and vision tasks.