Data-Free Distillation

Data-Free Distillation is a knowledge distillation technique that works without access to the original training data — using the teacher model itself to generate synthetic training data, or leveraging statistics stored in the teacher's batch normalization layers to guide data synthesis.

How Does Data-Free Distillation Work?

• Generator: Train a generator network to produce images that maximize the teacher's output diversity.
• BN Statistics: Use the running mean and variance stored in BatchNorm layers as targets for synthetic data statistics.
• Adversarial: Generate data that is hard for the student but easy for the teacher -> maximally informative.
• No Real Data: The entire distillation happens with synthetic data only.

Why It Matters

• Privacy: Original training data may be confidential, proprietary, or deleted after teacher training.
• Practical: Many deployed models have no associated training data pipeline available for re-training.
• Regulation: GDPR and similar regulations may prohibit retaining training data.

Data-Free Distillation is extracting knowledge without the textbook — training a student using only the teacher model itself, when the original training data is unavailable.