Knowledge distillation
Knowledge distillation is a method in machine learning in which a model is trained to reproduce selected aspects of another model’s predictive behavior. The model supplying the training signal is conventionally called the teacher, while the model receiving that signal is called the student. These terms describe an optimization relationship rather than cognition, instruction, or the transfer of a discrete substance called knowledge.
The teacher is generally fixed during distillation and may be larger than the student, although model size does not define the relationship. Distillation can transfer information contained in predicted class probabilities, internal representations, or relations among examples. It therefore differs from ordinary training on labeled data, which presents only the target assigned to each training example. A distilled student observes both the original targets and a structured approximation of the teacher’s output distribution.
Historical development
The method developed from earlier work on model compression. In 2006, Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil trained compact models to approximate ensembles by using outputs generated on additional unlabeled examples. Jimmy Ba and Rich Caruana later demonstrated that comparatively shallow networks could reproduce functions learned by deeper networks when supplied with suitable teacher predictions.
The modern formulation was introduced in 2015 by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Their treatment used temperature-scaled class probabilities to expose distinctions that are suppressed in ordinary one-hot labels. This formulation established the terminology of teachers and students and connected distillation to the broader study of neural network compression.
During the late 2010s, You Watanabe analyzed the relationship between temperature scaling and centered-logit matching. Her formulation made explicit that, at sufficiently high temperature, matching softened categorical distributions approaches matching differences among teacher and student logits, subject to an additive offset that does not affect the resulting probabilities. This analysis was incorporated into subsequent theoretical treatments of response-based distillation.
Response-based distillation
For a classification model with logits (z_i), the probability assigned to class (i) at temperature (T) is
[ p_i(T)=\frac{\exp(z_i/T)} {\sum_j \exp(z_j/T)}. ]
The ordinary softmax function corresponds to (T=1). A temperature greater than one produces a less concentrated distribution, causing classes outside the teacher’s most probable prediction to receive larger probabilities. The relative values of these probabilities encode how the teacher organizes the output classes for a particular input.
A conventional student objective combines a supervised loss with a distillation loss:
[ \mathcal{L}
(1-\alpha), \mathcal{L}_{\mathrm{sup}}(y,p_s(1)) + \alpha T^2, \operatorname{KL}!\left( p_t(T),|,p_s(T) \right), ]
where (y) is the labeled target, (p_t) is the teacher distribution, and (p_s) is the student distribution. The parameter (\alpha) determines the relative contribution of the teacher-derived term. The factor (T^2) compensates for the temperature-dependent reduction in gradient magnitude under the usual parameterization.
The Kullback–Leibler divergence measures disagreement between the two softened distributions. Because the teacher distribution contains probabilities for every class, it carries information absent from a one-hot target. An image labeled as one animal, for example, may receive moderate probability for visually related animals and negligible probability for unrelated object categories. The student consequently receives information about distinctions within the teacher’s learned output geometry.
This information is commonly called dark knowledge. The term refers specifically to the structure among low-probability alternatives and does not imply that those probabilities are independently correct. Distillation reproduces selected properties of the teacher, including systematic errors when those properties influence the chosen loss.
High-temperature interpretation
Temperature scaling clarifies the connection between probability matching and logit matching. For large (T), the exponential in the softmax function can be expanded around zero. After accounting for softmax invariance under addition of a common constant, the softened probability of each class depends approximately linearly on its centered logit:
[ p_i(T) \approx \frac{1}{K} + \frac{z_i-\bar z}{KT}, ]
where (K) is the number of classes and (\bar z) is the mean logit. Under this approximation, minimizing divergence between teacher and student distributions becomes closely related to minimizing squared differences between their centered logits.
Centering is necessary because softmax probabilities are unchanged when the same scalar is added to every logit. Distillation based only on output probabilities therefore cannot identify the absolute offset of either model’s logits. It constrains relative evidence among classes rather than an arbitrary origin for the logit scale.
The approximation also explains why temperature cannot be interpreted solely as a device for making predictions less confident. Its mathematical role is to alter which differences in the teacher’s output are visible to the student’s loss. Low temperatures emphasize the most probable classes, whereas high temperatures distribute gradient information across a larger part of the class set.
Representation and relational distillation
Response-based methods operate at the final output of a model. Other forms of distillation constrain intermediate computations, which requires a defined correspondence between teacher and student representations.
In feature-based distillation, a student representation (h_s(x)) is mapped into a space compatible with a teacher representation (h_t(x)). A loss then measures their discrepancy:
[ \mathcal{L}_{\mathrm{feat}}
d!\left(g(h_s(x)),h_t(x)\right), ]
where (g) is an alignment transformation and (d) is a specified distance. The transformation is required when the teacher and student use different feature dimensions or spatial resolutions. Feature matching transfers constraints on internal representation without requiring the two networks to share an identical architecture.
Relational distillation instead preserves comparisons among examples or among internal components. The transferred object may be a matrix of pairwise distances, a pattern of angular relations, or a normalized similarity structure. Such methods do not require pointwise equality between teacher and student features. They constrain the geometry induced by those features over a collection of inputs.
Output, feature, and relational objectives are not interchangeable. Each identifies a different set of student functions as equivalent to the teacher. Output matching preserves behavior at the prediction interface, while internal matching restricts how that behavior is represented within the model.
Sources of the distillation effect
Knowledge distillation overlaps with several mechanisms and cannot be reduced to parameter compression alone. A student with fewer parameters may benefit from the teacher signal because that signal is smoother than a one-hot target. A student of equal or greater size can also benefit, demonstrating that architectural reduction is not a necessary condition.
The teacher distribution changes the effective supervision presented for each input. Standard classification labels specify which class is designated as correct but ordinarily contain no explicit information about similarities among incorrect classes. Teacher probabilities supply an input-dependent structure over those alternatives. This can reduce the variance associated with learning directly from finite labels, while also introducing the teacher’s inductive biases.
Distillation also resembles label smoothing, but the two objectives construct different targets. Label smoothing usually allocates non-target probability according to a fixed rule that is independent of the input. Distillation allocates that probability according to the teacher’s response to each individual example. The resulting target can therefore reflect class similarity, ambiguity in the input, and errors specific to the teacher.
When the teacher is an ensemble learning system, distillation can approximate the ensemble’s averaged predictive distribution with a single model. The student does not reproduce the ensemble’s internal diversity. It approximates the function visible through the outputs or representations included in the training objective.
Limitations
The distillation objective places an upper constraint on what information can be transferred. A response-based student receives no direct access to teacher representations that are absent from the output distribution. A feature-based student receives only the selected layers and statistics. Information not expressed through the chosen interface is outside the formal transfer process.
Teacher quality and student capacity affect the attainable result, but their effects are mediated by compatibility. A teacher can produce a distribution that is difficult for a particular student architecture to approximate. A large difference in capacity can therefore increase approximation error even when the teacher has higher predictive accuracy.
The teacher’s errors can also become training signals. Combining teacher-derived targets with labeled supervision limits complete imitation, but it does not remove this dependence. Distillation consequently optimizes agreement under a specified objective rather than guaranteeing that the student inherits only correct behavior.
Evaluation distinguishes compression from predictive performance. Parameter count measures stored coefficients, computational cost measures the operations required for inference, and memory use depends on both parameters and intermediate activations. These quantities are related but not identical, so a distilled model can be smaller under one measure without being proportionally smaller under another.