What Is Model Distillation
Model distillation is a technique for training a smaller, faster "student" AI model to mimic the behavior of a larger, more capable "teacher" model, transferring much of the teacher's learned knowledge into a compact form that can run far more efficiently, often with only a modest reduction in accuracy.
Distillation works because a large model's own outputs carry richer information than the raw training labels alone. Rather than training the smaller student model only on the same basic correct-answer labels the teacher was originally trained on, distillation typically has the student learn from the teacher's full output patterns — including how confident it is across various possible answers — a richer training signal that helps the smaller model learn more efficiently than training from scratch on the original data alone.
| Aspect | Teacher model | Distilled student model |
|---|---|---|
| Size | Large | Much smaller |
| Accuracy | Highest achievable | Slightly lower, often close |
| Compute cost to run | High | Significantly lower |
| Typical role | Training-time reference | Deployed in production, especially resource-constrained settings |
Distillation and quantization solve related but distinct problems, and are frequently used together for maximum efficiency. Quantization reduces the numerical precision of an existing model's weights, while distillation trains an entirely new, smaller model architecture to mimic a larger one's behavior — these are complementary techniques, and a heavily optimized deployed model often uses both: first distilled down to a smaller architecture, then further quantized to reduce its numerical precision.
A well-executed distillation can preserve a surprising amount of the teacher model's practical capability despite a dramatic reduction in size. Because the student model learns from the teacher's nuanced output patterns rather than starting from nothing, a properly distilled student can often retain most of the teacher's performance on the tasks it was distilled for, even at a small fraction of the parameter count — though the student typically doesn't match the teacher's performance on tasks or edge cases well outside what it was specifically trained to handle.
Distillation is a major reason many AI products can serve fast, affordable responses to enormous numbers of users despite being built on top of far larger underlying models. Running a full-size, maximally capable model for every single user request would often be prohibitively slow and expensive at large scale, so many production AI systems rely on a distilled, more efficient version of a larger model for everyday use — reserving the full, larger model for training or for cases where its extra capability is genuinely needed.
Read model distillation through a teacher-and-student lens: rather than training a small model from scratch and hoping it learns as much as a much larger model did, distillation has the smaller model learn directly from a more capable one's behavior, transferring much of that hard-won capability into a form efficient enough to actually deploy at scale.