Feature Matching Distillation

Feature Matching Distillation (FitNets) is a knowledge distillation approach where the student is trained to match the teacher's intermediate feature representations — not just the final output, providing deeper knowledge transfer from the teacher's internal representations.

How Does Feature Matching Work?

• Hint Layers: Select intermediate layers from teacher and student.
• Projection: If dimensions differ, use a learnable linear projection ($W_s \cdot F_{student} \approx F_{teacher}$).
• Loss: L2 distance between projected student features and teacher features at matched layers.
• Paper: Romero et al., "FitNets: Hints for Thin Deep Nets" (2015).

Why It Matters

• Deeper Transfer: Transfers knowledge from internal representations, not just output predictions.
• Thin & Deep: Enables training very deep, thin student networks that would otherwise be difficult to train.
• Layer Matching: The choice of which teacher and student layers to match significantly impacts performance.

Feature Matching Distillation is transferring the teacher's internal thought process — teaching the student to think like the teacher at every level, not just arrive at the same answer.