Teaching Assistant

Teaching Assistant (TA) in knowledge distillation is a technique that introduces an intermediate-sized model between a very large teacher and a very small student — bridging the capacity gap that causes direct distillation to fail when the teacher is too powerful relative to the student.

How Does TA Work?

• Problem: When the capacity gap between teacher and student is too large, the student cannot effectively learn from the teacher's complex output distribution.
• Solution: Train an intermediate "teaching assistant" model from the teacher first, then use the TA to train the final student.
• Chain: Teacher -> TA -> Student. Each step has a manageable capacity gap.
• Paper: Mirzadeh et al., "Improved Knowledge Distillation via Teacher Assistant" (2020).

Why It Matters

• Bridging the Gap: A ResNet-110 teacher may not distill well to a ResNet-8 student directly. A ResNet-32 TA bridges the gap.
• Multi-Step: Multiple TAs can be chained for very large capacity gaps.
• Practical: Important when the deployment target has extremely limited resources.

Teaching Assistant is the bridge between master and novice — an intermediate model that translates expert knowledge into a form that a small student can actually absorb.