Self-Distillation

Self-distillation is the training strategy where a model learns from softened targets produced by another instance of itself or an exponential moving average teacher - in vision transformers, it improves calibration, representation smoothness, and low-label transfer.

What Is Self-Distillation?

• Definition: Student model matches probabilistic outputs or features from a teacher derived from the same architecture family.
• Teacher Sources: EMA teacher, previous epoch checkpoint, or stronger pretrained variant.
• Target Type: Logits, intermediate features, or token-level distributions.
• Objective Blend: Distillation loss is combined with supervised or self-supervised base loss.

Why Self-Distillation Matters

• Generalization Gain: Soft targets carry richer uncertainty information than hard labels.
• Calibration Improvement: Reduces extreme confidence spikes in predictions.
• Stability: Teacher signal regularizes optimization, especially in deep models.
• Label Efficiency: Helps student perform better with fewer labeled examples.
• Compression Path: Supports transfer from large teacher to smaller deployment models.

Distillation Configurations

Logit Distillation:
- Student matches teacher class probability distribution.
- Temperature controls softness of targets.

Feature Distillation:
- Align intermediate representations between teacher and student.
- Useful for dense tasks and architecture transfer.

Token Distillation:
- Match patch-level outputs for richer spatial guidance.
- Effective in ViT pipelines.

Implementation Guidance

• Temperature Tuning: Too sharp targets reduce benefit, too soft can dilute signal.
• Loss Weighting: Balance distillation and base losses across training stages.
• Teacher Stability: EMA teachers often provide smoother supervision.

Self-distillation is a high-impact regularization and transfer mechanism that lets models learn from structured soft supervision instead of only hard targets - it is a core ingredient in modern ViT training recipes.