Selective Knowledge Distillation

Selective Knowledge Distillation is a distillation approach that carefully chooses which knowledge to transfer from teacher to student — rather than blindly mimicking all teacher outputs, selectively transferring only the most informative or relevant knowledge for the student's capacity.

How Does Selective KD Work?

• Sample Selection: Focus on hard or informative samples where the teacher's guidance is most valuable.
• Channel Selection: Transfer only the most important feature channels, not all intermediate representations.
• Class Selection: For many-class problems, distill from the top-k most relevant classes only.
• Confidence-Based: Weight the distillation loss by teacher's confidence — focus on samples where teacher is most certain.

Why It Matters

• Efficiency: Not all teacher knowledge is equally useful for the student. Selective transfer avoids noise.
• Capacity Match: A small student may not have capacity to absorb everything — selective KD prioritizes.
• Performance: Often outperforms full distillation by reducing the "noise" of irrelevant teacher signals.

Selective Knowledge Distillation is curated mentoring — choosing the most important lessons to teach rather than overwhelming the student with everything.