Knowledge Distillation: How Hinton, Vinyals, and Dean Taught Small Models to Learn from Soft Targets

Training a large neural network and deploying it at scale are two fundamentally different engineering problems. During training, the goal is to extract as much signal as possible from data, which favors large models, long compute budgets, and ensemble strategies that average predictions from many independently trained networks. During deployment, the goal is to serve predictions quickly and cheaply to many users, which favors small models with low memory footprints and fast inference. For most of the 2010s, these two goals pulled in opposite directions, and practitioners chose a point on the tradeoff curve and accepted its costs. The paper 'Distilling the Knowledge in a Neural Network,' written by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean at Google, submitted to arXiv on March 9, 2015 (arXiv:1503.02531) and presented at the NIPS 2014 Deep Learning Workshop, introduced a training procedure called knowledge distillation that substantially narrows this gap. The core insight is that a trained model's output probability distribution, not just its hard classification decision, encodes a rich description of how the model generalizes. If a small model can be trained to match that distribution rather than just the original labels, it learns to generalize in the same way as the large model while remaining small enough to deploy efficiently.

Soft targets and the information in wrong answers

A neural network trained for classification produces a probability distribution over all output classes. The training objective typically pushes most of that probability mass onto the correct class, with very small residual probabilities assigned to incorrect classes. These residual probabilities look like noise, but the paper argues they carry structured information about how the model generalizes. The paper offers a concrete example: an image of a BMW might have a very small probability of being misclassified as a garbage truck, but that probability is still many orders of magnitude larger than the probability of being misclassified as a carrot. The ratio of those small probabilities encodes a learned similarity structure: the model has concluded that trucks and cars share more features than cars and vegetables do. A student model trained on one-hot labels cannot access this information. It only sees 'this image is a BMW' and learns nothing about the model's internal representation of the relationship between BMW images and other vehicle classes. A student trained on the teacher's soft targets sees the full probability distribution and learns that relationship directly.

Temperature scaling and the distillation mechanism

The technical problem with using soft targets directly is that a well-trained large model tends to assign probabilities very close to one for the correct class and probabilities very close to zero for incorrect classes, particularly on tasks like MNIST where the training signal is clean and abundant. A soft target of 0.9999 for the correct class and 0.0001 distributed across the wrong classes is barely more informative than a hard one-hot label. The paper introduces temperature scaling to address this. The standard softmax function computes the probability for class i as the exponential of logit z_i divided by the sum of exponentials over all logits. Distillation generalizes this by dividing each logit by a temperature parameter T before applying the softmax. When T equals one, this is the standard softmax. When T is greater than one, the distribution becomes softer, spreading probability mass more evenly across classes and making the relative similarities between incorrect classes more visible. The paper uses T greater than one during training to generate soft targets from the teacher and to train the student. At inference time, T is set back to one. The specific value of T is a hyperparameter; the paper reports that values in the range of 2 to 5 work well on the tasks evaluated, with the optimal value depending on the task and the relative sizes of the teacher and student.

The training objective: combining soft and hard targets

The student model is trained on a weighted combination of two loss terms. The first is the standard cross-entropy loss against the true hard labels, computed at temperature T equal to one. The second is the distillation loss, which is the cross-entropy between the student's soft output at temperature T and the teacher's soft output at the same temperature T. The paper recommends weighting the hard-label loss with a small coefficient and giving the distillation loss the larger weight, because the soft targets already contain the information in the hard labels and carry additional relational structure on top of it. Multiplying the distillation loss by T squared is recommended to keep the magnitude of the gradient contributions roughly balanced as T varies, because the gradient of the soft cross-entropy loss scales as one over T squared relative to the hard-label loss. This weighting is a practical detail rather than a theoretical requirement, but it matters empirically.

MNIST, speech recognition, and the JFT specialist ensemble

The paper reports three sets of experiments. The MNIST experiment demonstrates the concept in its cleanest form: a large model distilled into a much smaller model achieves error rates very close to the teacher ensemble. The paper notes that the distilled student achieves surprising results because it has generalized in the same way as the teacher rather than in the way the small model's own architecture would naturally generalize when trained from scratch. The more practically significant experiment applies distillation to speech recognition. The teacher is an ensemble of acoustic models trained on a large commercial dataset. Distillation produces a single model that significantly improves over a single model trained directly on the same data, closing most of the gap between a single model and the full ensemble. The paper describes this as improvement in a heavily used commercial system, but does not report precise word error rate numbers in the arXiv version. The third experiment introduces a specialist ensemble designed to handle the JFT dataset, an internal Google dataset with approximately 15,000 classes. The full generalist model struggles to distinguish fine-grained classes within broad categories. The paper trains specialist models, each focused on a confusing subset of classes, and uses distillation to combine their knowledge. The specialists are trained in parallel and rapidly, unlike standard ensemble members, because each specialist only needs to distinguish a small set of classes.

• Hinton, Vinyals, Dean (arXiv:1503.02531, submitted March 9, 2015): NIPS 2014 Deep Learning Workshop. All three authors affiliated with Google Inc. at time of publication.
• Core mechanism: train student to match teacher's temperature-scaled softmax output (soft targets) rather than one-hot hard labels.
• Temperature T greater than one softens the output distribution, surfacing relative probabilities of incorrect classes and making inter-class similarity structure visible to the student.
• Training loss: weighted sum of hard-label cross-entropy (T = 1) and distillation cross-entropy (T > 1). Distillation term weighted by T squared to balance gradient magnitudes.
• MNIST: distilled small model achieves error close to the teacher ensemble, outperforming a small model trained directly on hard labels.
• Speech recognition: distilled single model significantly improves over a baseline single model, nearly matching the teacher ensemble on an internal commercial system.
• JFT (approximately 15,000 classes): specialist ensemble approach trains models in parallel to focus on confusable fine-grained subsets; distillation aggregates their knowledge.
• The term 'dark knowledge' refers informally to the information about inter-class relationships encoded in the teacher's soft probability distribution, which is invisible in hard one-hot labels.

Influence on modern model compression and LLM training

The 2015 paper established knowledge distillation as a standard technique in the model compression toolkit, and the technique has been applied and extended in numerous directions since. Relation-based distillation methods, proposed in subsequent work, extended the core idea from matching output probabilities to matching intermediate layer activations, attention maps, or feature similarity matrices between teacher and student. In the era of very large language models, distillation became a primary strategy for producing smaller deployable variants: GPT-2 led to DistilGPT-2, BERT led to DistilBERT (which retained roughly 97% of BERT's performance on GLUE while being 40% smaller and 60% faster), and similar approaches have been applied to models at scales from hundreds of millions to hundreds of billions of parameters. The intuition from the original paper, that soft targets contain more information per training example than hard labels, has been confirmed empirically across a wide range of settings. One underappreciated consequence is that distillation can reduce the amount of labeled data required to train a capable student: because the soft targets carry richer supervision signal, the student learns effectively from fewer examples than would be needed if it were trained from scratch on hard labels. PromptingIndex covers knowledge distillation alongside related model compression topics including quantization (GPTQ, QLoRA, BitNet), efficient fine-tuning (LoRA, GaLore), and inference optimization (speculative decoding, PagedAttention).