Mean Teacher

Mean Teacher is a semi-supervised learning method that maintains an exponential moving average (EMA) of model weights as the "teacher" — the student model is trained on labeled data, while a consistency loss encourages the student to match the teacher's predictions on unlabeled data.

How Does Mean Teacher Work?

• Student: Trained with gradient descent on labeled loss + consistency loss.
• Teacher: $\theta_{teacher} = \alpha \cdot \theta_{teacher} + (1-\alpha) \cdot \theta_{student}$ (EMA of student weights, $\alpha \approx 0.999$).
• Consistency: $\mathcal{L}_{cons} = f_{teacher}(x + \text{noise}) - f_{student}(x + \text{noise'}) ^2$.
• Paper: Tarvainen & Valpola (2017).

Why It Matters

• Stable Targets: The EMA teacher produces more stable and accurate targets than the student alone.
• Foundation: Inspired BYOL (self-supervised), EMA-based methods in detection, and modern SSL frameworks.
• Effective: Significant improvement over Temporal Ensembling with negligible additional compute.

Mean Teacher is the smoothed mentor — an exponentially averaged version of the model that provides stable, high-quality targets for semi-supervised learning.