Teacher-Student Framework

Teacher-Student Framework is the general paradigm where a pre-trained "teacher" model guides the training of a "student" model — the teacher provides soft targets, intermediate features, or other supervision signals that help the student learn better than it could from data alone.

What Is the Teacher-Student Framework?

• Teacher: Large, accurate, pre-trained model (or an ensemble). Fixed during distillation.
• Student: Smaller, efficient model to be deployed. Trained to mimic the teacher.
• Supervision: Teacher's soft outputs (KD), features (FitNets), attention maps, or relational structure.
• Applications: Model compression, SSL (DINO), semi-supervised learning, domain adaptation.

Why It Matters

• Universal Pattern: The teacher-student paradigm appears across model compression, self-supervised learning, and semi-supervised learning.
• Flexibility: The teacher can be a larger model, an ensemble, or even the same model at a different training stage (self-distillation).
• Deployment: Enables deploying compact, fast models that retain the accuracy of much larger ones.

Teacher-Student Framework is the master-apprentice relationship of deep learning — the universal pattern of knowledge transfer from a capable model to a practical one.