Knowledge Distillation
Knowledge distillation is a machine learning technique where a smaller, more efficient model is trained to mimic the outputs of a larger, more capable model, transferring much of the larger model's learned behavior into a form that's far cheaper to run.
Knowledge distillation exists because the largest, most capable models are often too slow and expensive to run in many real-world settings, but training a smaller model from scratch to match that same capability directly is genuinely difficult. Rather than training a small model purely from the original raw data and hoping it independently discovers similarly good behavior, knowledge distillation instead trains that smaller student model specifically to mimic the outputs of an already-trained, larger teacher model, transferring much of the teacher's learned behavior into a smaller, cheaper form more directly and effectively than training from scratch typically achieves.
| Aspect | Training from scratch | Knowledge distillation |
|---|---|---|
| Training target | Raw labels only | Teacher model's output behavior |
| Typical resulting capability | Less predictable | Often closer to teacher's capability |
| Model size after training | Depends on design choice | Deliberately kept small |
| Common use | Baseline model development | Compressing a large model for deployment |
Knowledge distillation typically trains the student not just to match the teacher's final answer, but to match the teacher's full output distribution, which carries additional useful information about relative confidence between different possible answers. Rather than only training the student to reproduce the teacher's single top answer, distillation commonly has the student learn to match the teacher's full probability distribution across possible outputs, since that fuller distribution carries additional signal about how confidently the teacher favored one answer over close alternatives.
Knowledge distillation is one of several distinct techniques used to make large models more practical to deploy, alongside quantization and pruning, each addressing model efficiency in a different way. Because reducing a model's deployment cost can be approached from multiple angles, knowledge distillation, which produces a genuinely smaller model architecture, is often used alongside other techniques like quantization or pruning, which reduce cost differently, and these approaches are sometimes combined for even greater efficiency gains.
Distilled student models don't always fully match their teacher's capability, and the resulting capability gap depends significantly on how much smaller the student model is relative to the teacher. Because compressing knowledge into a meaningfully smaller model inevitably involves some loss of capability, the practical question in any distillation effort is how much smaller the student can be made while still retaining acceptable performance for the intended use case, a tradeoff that has to be evaluated for each specific application.
Read knowledge distillation through an apprentice lens: rather than having a new apprentice learn a craft entirely from raw first principles, having them closely observe and imitate an experienced master's actual work transfers much of that expertise more directly, producing a less experienced but still highly capable worker far faster than starting from nothing.