Knowledge Distillation Variants

Knowledge Distillation Variants are extensions of the original Hinton et al. (2015) teacher-student distillation framework — encompassing different ways to transfer knowledge from a larger model to a smaller one, including response-based, feature-based, and relation-based approaches.

Major Variants

• Response-Based: Student mimics teacher's soft output probabilities (original KD). Loss: KL divergence on softened logits.
• Feature-Based (FitNets): Student mimics teacher's intermediate feature representations. Requires projection layers for dimension matching.
• Relation-Based (RKD): Student preserves the relational structure (distances, angles) between samples as computed by the teacher.
• Attention Transfer: Student mimics teacher's attention maps (spatial or channel attention).

Why It Matters

• Flexibility: Different variants are optimal for different architectures and tasks.
• Complementary: Multiple distillation signals can be combined for stronger compression.
• Scale: Used to compress billion-parameter LLMs into practical deployment-sized models.

Knowledge Distillation Variants are the different channels of knowledge transfer — each capturing a different aspect of what the teacher model knows.