Temperature in Distillation

Temperature in Distillation is the softmax scaling parameter $\tau$ used to control the smoothness of the teacher's output distribution — higher temperature produces softer probabilities that reveal more dark knowledge, while lower temperature produces sharper, more confident distributions.

How Does Temperature Work?

• Softmax: $p_i = \frac{\exp(z_i / \tau)}{\sum_j \exp(z_j / \tau)}$
• $\tau = 1$: Standard softmax. One class dominates.
• $\tau = 5-20$: Softer distribution. Non-dominant classes become visible.
• $\tau \rightarrow \infty$: Uniform distribution (maximum entropy).
• Training: Both teacher and student use the same $\tau$ during distillation.

Why It Matters

• Information Extraction: Higher $\tau$ extracts more dark knowledge from the teacher's logits.
• Typical Values: $\tau = 3-10$ works well in practice. Too high dilutes the signal.
• Scaling: The distillation loss is multiplied by $\tau^2$ to maintain gradient magnitude across temperatures.

Temperature is the zoom lens on dark knowledge — adjusting how much inter-class similarity information is exposed from the teacher's output distribution.