Online Distillation

Online Distillation is a knowledge distillation approach where teacher and student networks are trained simultaneously — rather than the traditional offline approach where the teacher is pre-trained and fixed. Both networks learn from each other during training.

How Does Online Distillation Work?

• Mutual Learning (DML): Two networks are trained in parallel. Each one uses the other's soft predictions as additional supervision.
• Co-Distillation: Multiple models exchange knowledge during training rounds.
• ONE (One-for-all): A single multi-branch network where branches distill knowledge to each other.
• No Pre-Training: Unlike offline KD, no separate teacher training phase is needed.

Why It Matters

• Efficiency: Eliminates the expensive pre-training phase for the teacher model.
• Mutual Benefit: Both networks improve from the knowledge exchange — even models of the same size benefit.
• Ensemble Effect: The aggregated knowledge from multiple online students often exceeds any single model.

Online Distillation is collaborative learning between networks — where models teach each other simultaneously, improving together without a pre-trained teacher.