Layer Distillation

Layer distillation (also called hint-based or intermediate distillation) trains student networks by matching intermediate layer representations to teacher layers, not just final outputs, providing richer supervision for knowledge transfer. Motivation: matching only output logits loses intermediate computational structure; internal representations encode useful patterns. Hint layers: designated teacher layer outputs that guide corresponding student layers—student minimizes distance to teacher's intermediate features. Loss function: L = L_task + λ × Σ_l T_l(f_teacher^l) - f_student^l ², where T_l is optional transformation (handle dimension mismatch). FitNets: foundational paper introducing thinner, deeper students trained with hint layers. Layer mapping: which teacher layers correspond to which student layers—typically match relative depth or stage outputs. Transformation layers: when teacher and student have different dimensions, add projector network to align representations. Attention transfer: distill attention maps (where model focuses) rather than raw feature values. Progressive distillation: sequentially match layers from shallow to deep during training. Feature distillation variants: relational distillation (preserve relationships between samples), contrastive distillation (negative samples). Benefits: (1) trains significantly thinner students effectively, (2) captures structural knowledge beyond output predictions, (3) enables deeper small models. Applications: efficient deployment, multi-stage distillation pipelines. Core technique for compression when strong teacher guidance is desired.