Self-Distillation

Self-Distillation is a knowledge distillation technique where the teacher and student share the same architecture — the model distills knowledge into itself, either by using a deeper version as teacher, using earlier training checkpoints, or distilling from the full model into auxiliary classifiers at intermediate layers.

How Does Self-Distillation Work?

• Same Architecture: Teacher and student have identical structure (unlike traditional KD where teacher is larger).
• Variants:
• Born-Again Networks: Train student = teacher architecture on teacher's soft labels.
• DINO: EMA teacher provides targets for the student (self-distillation with momentum).
• Intermediate Classifiers: Auxiliary classifiers at hidden layers distill from the final classifier.
• Surprise: Self-distilled models often outperform the original teacher!

Why It Matters

• Free Performance: Improves accuracy without increasing model size or changing architecture.
• Label Smoothing Effect: Soft targets provide richer training signal than hard labels.
• Foundation Models: DINO and DINOv2 are fundamentally self-distillation frameworks.

Self-Distillation is the student becoming the teacher — a model improving itself by learning from its own refined outputs.