Temperature Scaling

What It Is

Temperature scaling divides a neural network’s raw logits by a scalar T before applying softmax. At T=1 you get the standard softmax. At T>1 the output distribution becomes softer (more spread out, higher entropy). At T<1 the distribution becomes sharper (more confident, closer to one-hot).

Why It Matters

Temperature is the key control knob in knowledge distillation: raising T reveals the structural similarity information buried in a teacher model’s near-zero probabilities. It’s also used for calibration (adjusting model confidence to match actual accuracy) and for controlling sampling diversity at inference time.

How It Works

$q_i=\frac{e^{z_i/T}}{{\sum }_j{e}^{z_j/T}}$

As T → ∞, all $q_i$ approach 1/N (uniform). As T → 0, all probability collapses to the argmax. For distillation, T=3–10 is typical: dominant class still wins, but runner-ups reveal similarity structure.

Key Sources

• knowledge-distillation-hinton

• emergent-abilities-of-large-language-models

• gpt-4-technical-report

• language-models-are-few-shot-learners

• language-models-are-unsupervised-multitask-learners

• scaling-laws-for-neural-language-models

• scaling-laws-neural-language-models

• switch-transformer-sparse-mixture-of-experts

• training-compute-optimal-large-language-models

Related Concepts

• distillation — temperature scaling enables soft targets in knowledge distillation
• sampling — temperature controls diversity in LLM generation