Multi-Teacher Distillation

Multi-Teacher Distillation is a knowledge distillation approach where a single student learns from multiple teacher models simultaneously — combining knowledge from diverse teachers that may have different architectures, training data, or areas of expertise.

How Does Multi-Teacher Work?

• Aggregation: Teacher predictions are combined by averaging, weighted averaging, or learned attention.
• Specialization: Different teachers may specialize in different classes or domains.
• Loss: $\mathcal{L} = \mathcal{L}_{CE} + \sum_t \alpha_t \cdot \mathcal{L}_{KD}(student, teacher_t)$
• Ensemble-Like: The student effectively distills the knowledge of an ensemble into a single model.

Why It Matters

• Diversity: Multiple teachers provide diverse perspectives, reducing bias and improving generalization.
• Ensemble Compression: Compresses an ensemble of large models into one small model for deployment.
• Multi-Domain: Teachers trained on different domains contribute complementary knowledge.

Multi-Teacher Distillation is learning from a panel of experts — absorbing diverse knowledge from multiple specialists into a single efficient model.