Mutual Learning

Mutual Learning is a collaborative training strategy where two or more networks train simultaneously and teach each other — each network uses the other's soft predictions as an additional supervisory signal, improving both models beyond what either could achieve alone.

How Does Mutual Learning Work?

• Setup: Two (or more) networks with the same or different architectures, trained on the same data.
• Loss: Each network optimizes: $\mathcal{L} = \mathcal{L}_{CE} + \alpha \cdot D_{KL}(p_1 p_2)$ (and vice versa).
• No Pre-Training: Unlike traditional KD, no pre-trained teacher is needed.
• Paper: Zhang et al., "Deep Mutual Learning" (2018).

Why It Matters

• Mutual Improvement: Even two identical networks improve each other through mutual learning (surprising result).
• Ensemble Effect: Each network benefits from the regularizing effect of the other's predictions.
• Efficiency: Achieves distillation benefits without the cost of pre-training a large teacher model.

Mutual Learning is peer tutoring for neural networks — two models learning together and teaching each other, achieving better results than studying alone.