Neural Network Distillation Online

Online Distillation and Co-Distillation is the training paradigm where multiple neural networks teach each other simultaneously during training — unlike traditional knowledge distillation where a pre-trained large teacher transfers knowledge to a smaller student, online distillation trains teacher and student (or multiple peers) jointly from scratch, enabling mutual improvement where networks with different architectures or capacities share complementary knowledge through soft label exchange, logit matching, and feature alignment without requiring a separately trained teacher model.

Traditional vs. Online Distillation

Traditional (Offline) Distillation:
  Step 1: Train large teacher to convergence
  Step 2: Freeze teacher → train student on teacher's soft labels
  Cost: 2× training time (teacher + student)

Online (Co-)Distillation:
  Step 1: Train all networks simultaneously
  Each network is both teacher AND student
  Cost: ~1.3× training a single network (parallel)

Key Approaches

Method	Mechanism	Networks	Key Idea
Deep Mutual Learning (DML)	Logit-based KL loss between peers	2+ peers	Peers teach each other
Co-Distillation	Feature + logit exchange	2+ models	Different architectures share knowledge
Self-Distillation	Model teaches itself across layers	1 model	Deeper layers teach shallower layers
Born-Again Networks	Sequential self-distillation	1 → 1 → 1	Student matches or beats teacher
ONE (Online Ensemble)	Shared backbone + multiple heads	1 backbone	Gate network selects ensemble teacher

Deep Mutual Learning

# Two networks training together
for batch in dataloader:
    logits_1 = model_1(batch)
    logits_2 = model_2(batch)
    
    # Standard CE loss for both
    loss_ce_1 = cross_entropy(logits_1, labels)
    loss_ce_2 = cross_entropy(logits_2, labels)
    
    # Mutual KL divergence (each teaches the other)
    loss_kl_1 = kl_div(log_softmax(logits_1/T), softmax(logits_2/T)) * T*T
    loss_kl_2 = kl_div(log_softmax(logits_2/T), softmax(logits_1/T)) * T*T
    
    # Combined losses
    loss_1 = loss_ce_1 + alpha * loss_kl_1
    loss_2 = loss_ce_2 + alpha * loss_kl_2

Why Does Mutual Learning Work?

Different random initializations → different local features learned.
Each model discovers patterns the other missed → knowledge complementarity.
Soft labels provide richer training signal than hard one-hot labels.
Dark knowledge: The relative probabilities of incorrect classes carry information about data structure.
Result: Both models end up better than either would alone — even equally-sized peers improve each other.

Self-Distillation

Add auxiliary classifiers at intermediate layers.
Deep layers' soft predictions train shallow layers.
At inference, use only the final layer (no overhead).
Surprisingly: Even the deepest layer improves from teaching shallower ones.

Applications

Application	Benefit
Edge deployment	Train compressed model without pre-training teacher
Federated learning	Clients co-distill across communication rounds
Ensemble compression	Distill ensemble into single model during training
Continual learning	Old and new task models teach each other
Multi-modal training	Vision and language models co-distill

Online distillation is the efficient alternative to traditional teacher-student training — by eliminating the need for a separately pre-trained teacher and enabling networks to improve each other during joint training, co-distillation reduces total training cost while often achieving better accuracy than offline distillation, making it particularly valuable when training large teacher models is impractical or when mutual knowledge exchange between diverse model architectures is desired.

Neural Network Distillation Online

Take neural network distillation online further