Zero-Shot Distillation

Zero-Shot Distillation is a variant of data-free distillation where the student is trained without any real data or data generation process — relying entirely on the teacher's learned parameters and the structure of the output space to transfer knowledge.

How Does Zero-Shot Distillation Work?

• Crafted Inputs: Generate pseudo-data by optimizing random noise to maximize specific class activations in the teacher.
• Model Inversion: Use gradient-based optimization to "invert" the teacher — finding inputs that produce representative outputs.
• Dirichlet Sampling: Sample from the simplex of class probabilities to create diverse soft label targets.
• Difference from Data-Free: Zero-shot is even more restrictive — no generator network training, just direct optimization.

Why It Matters

• Extreme Constraint: When not even a generator can be trained (no compute budget for data generation).
• Model IP: Enables knowledge transfer from a black-box teacher API with minimal queries.
• Research: Explores the fundamental limits of how much knowledge can be extracted from a model without data.

Zero-Shot Distillation is knowledge transfer at the extreme — distilling a model's knowledge with literally zero training examples from any source.