Few-Shot Distillation

Few-Shot Distillation is a knowledge distillation approach that works with only a small number of labeled examples — combining the teacher's dark knowledge with data augmentation and meta-learning techniques to effectively train a student model from very limited data.

How Does Few-Shot Distillation Work?

• Setup: Very few labeled examples (1-10 per class) available for distillation.
• Teacher: Provides soft labels for the limited data + any augmented versions.
• Augmentation: Heavy data augmentation (CutMix, MixUp, RandAugment) to amplify the small dataset.
• Meta-Learning: Some approaches use meta-learning to optimize the distillation procedure itself.

Why It Matters

• Low-Resource: Many real-world applications have very limited labeled data for the target domain.
• Domain Shift: When the teacher was trained on domain A but the student needs to operate on domain B with few examples.
• Rapid Deployment: Enables quick model deployment in new domains without extensive data collection.

Few-Shot Distillation is learning from a teacher with almost no examples — maximizing knowledge transfer efficiency when data is extremely scarce.