Dark Knowledge

Dark Knowledge is the rich information contained in a teacher model's soft output distribution — the relative probabilities assigned to incorrect classes reveal the model's learned similarity structure, which is far more informative than the hard one-hot label.

What Is Dark Knowledge?

• Example: For an image of a cat, the teacher might output: cat=0.85, dog=0.10, fox=0.03, car=0.001.
• Information: The high probability for "dog" tells the student that cats and dogs look similar. "Car" being near-zero teaches they are unrelated.
• Hard Labels: Only say "cat." No information about similarity to other classes.
• Temperature: Higher temperature ($\tau$) softens the distribution, revealing more dark knowledge.

Why It Matters

• Richer Supervision: Dark knowledge provides orders of magnitude more information per training sample than hard labels.
• Generalization: Students trained on soft targets generalize better because they learn inter-class relationships.
• Foundation: The entire knowledge distillation framework is built on the insight that dark knowledge exists and is transferable.

Dark Knowledge is the hidden curriculum in a teacher's predictions — the subtle class-similarity information that hard labels completely discard.