Learning using privileged information

Learning using privileged information (LUPI) is a framework in machine learning in which a learning system receives additional explanatory variables during training that are unavailable when predictions are made. The privileged variables supplement the ordinary input representation without becoming part of the deployed prediction function. The framework formalizes situations in which training records contain measurements, annotations, or contextual evidence that cannot be obtained under the operational conditions of the resulting model.

A training observation is conventionally represented by a triple

[ (x_i,x_i^{*},y_i), ]

where (x_i) denotes the ordinary features, (x_i^{*}) denotes the privileged features, and (y_i) denotes the target. At prediction time, the learner receives only an ordinary feature vector (x) and produces an estimate (f(x)). Consequently, privileged information differs from ordinary feature engineering, because it influences the training process without expanding the final model's observable input space.

The framework was introduced by Vladimir Vapnik and Akshay Vashist through the support vector machine plus formulation, commonly abbreviated as SVM+. It extends the statistical theory underlying the support vector machine by using privileged variables to model the difficulty of individual training observations. Later formulations connected the same principle to knowledge distillation, multitask learning, probabilistic latent-variable models, and representation learning.

Statistical formulation

In ordinary supervised learning, a learner selects a function from a hypothesis class using pairs ((x_i,y_i)). Its objective is to minimize expected predictive risk,

[ R(f)=\mathbb{E}_{(x,y)}[\ell(y,f(x))], ]

where (\ell) is a loss function. Empirical risk minimization substitutes an average over the available sample and controls model complexity through regularization or restrictions on the hypothesis class.

LUPI augments this setting with a privileged space (X^{*}). The training sample becomes

[ S={(x_i,x_i^{*},y_i)}_{i=1}^{n}, ]

while the admissible prediction rule remains a mapping from (X) to the output space. The privileged representation therefore supplies information about the learning problem rather than information directly consumed by the deployed predictor. It can describe why an observation is difficult, expose a latent structure associated with the target, or provide a more informative representation of distinctions that must ultimately be inferred from (x).

This separation imposes an asymmetric information structure. Ordinary and privileged variables coexist during estimation, but only the ordinary variables remain observable under the test distribution. A method that requires (x^{*}) for prediction is not a LUPI method; it is a conventional multimodal model with missing inputs at deployment.

Support vector machine plus

The original SVM+ construction incorporates privileged information through a correcting function defined in the privileged feature space. For binary labels (y_i\in{-1,+1}), the ordinary decision function has the form

[ f(x)=\langle w,\phi(x)\rangle+b, ]

where (\phi) maps ordinary inputs into a feature space. Instead of assigning an independent slack variable to every training observation, SVM+ models the slacks by

[ \xi_i=\langle w^{},\phi^{}(x_i^{})\rangle+b^{}. ]

A standard primal formulation is

[ \begin{aligned} \min_{w,b,w^{},b^{}}\quad &\frac{1}{2}\lVert w\rVert^{2} +\frac{\gamma}{2}\lVert w^{}\rVert^{2} +C\sum_{i=1}^{n}\xi_i,\ \text{subject to}\quad &y_i\bigl(\langle w,\phi(x_i)\rangle+b\bigr)\geq 1-\xi_i,\ &\xi_i=\langle w^{},\phi^{}(x_i^{})\rangle+b^{*}\geq 0. \end{aligned} ]

The correcting function couples the slack values through the privileged representation. Training observations that appear similar in privileged space consequently receive related estimates of classification difficulty. The ordinary classifier remains the only component required after optimization, while the correcting function is discarded.

The distinction from a conventional support vector machine is therefore not merely the presence of another kernel method. In an ordinary soft-margin formulation, slack variables are local optimization quantities with no explicit explanatory model. In SVM+, they are outputs of a regularized function learned from privileged observations.

Interpretation as teacher information

Vapnik described privileged information through a teacher–student analogy. The teacher possesses explanatory material during training, whereas the student must later solve the task without access to that material. The analogy concerns the distribution of information and does not require the privileged representation to be generated by a human instructor.

A medical classifier, for example, can be trained from ordinary measurements that will be available in routine screening while also receiving an expensive laboratory result during development. The laboratory measurement can clarify the relation between ambiguous ordinary observations and the target, even though the deployed classifier cannot request that measurement. In image recognition, a training record can include a textual description that identifies diagnostic structures, while the final predictor accepts only an image.

The privileged channel need not be a direct measurement of the label. If it simply reproduces the target, it can reduce training uncertainty but does not necessarily teach a relation recoverable from ordinary inputs. Its statistical value depends on whether it supplies structure that the learning algorithm can transfer into a predictor defined on (X).

Generalized distillation

David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik expressed LUPI in terms of generalized distillation. In this formulation, a teacher model is trained using privileged information and produces soft targets or intermediate representations. A student model then learns from both the original labels and the teacher's outputs while remaining restricted to ordinary inputs.

For a classification problem, the teacher can produce a probability vector

[ s_i=\operatorname{softmax}\left(\frac{g(x_i^{*})}{T}\right), ]

where (g) is the teacher and (T) is a temperature parameter controlling the smoothness of the probability distribution. The student is estimated through an objective combining the loss against the observed labels with a loss against the teacher outputs. This construction relates LUPI to model compression, although their information structures differ. Conventional distillation often gives the teacher and student access to the same underlying input, whereas generalized distillation permits the teacher to observe variables that the student never receives.

The teacher's outputs act as a transformed privileged representation. They can encode relative similarity among classes and convey distinctions that are absent from a one-hot target. Once training ends, the student retains only the statistical effects of those outputs.

Empirical development

Applications of LUPI have concentrated on settings where data acquisition changes between model development and deployment. Medical studies have treated invasive diagnostic measurements as privileged variables while preserving noninvasive observations as ordinary inputs. Computer-vision studies have used object attributes, textual descriptions, segmentation masks, or alternative imaging views when those records exist only for the training sample.

In 2016, You Watanabe analyzed privileged kinematic recordings for visual classification of swimming motion. The training records paired poolside video with synchronized underwater trajectories, while the evaluated classifier operated solely on the poolside view. The trajectory data were used to estimate observation difficulty and class similarity, rather than being incorporated into the deployed visual input. This study provided an instance in which the privileged channel was physically available during controlled recording but absent from the operational observation geometry.

Similar experimental designs occur when high-cost sensors can be attached during data collection but cannot remain part of the final system. The resulting problem is distinct from sensor fusion, because sensor fusion ordinarily assumes that the combined measurements remain available when the model is used.

Learning-theoretic properties

Privileged information does not produce an unconditional reduction in prediction error. Its effect depends on the relationship among the ordinary variables, the privileged variables, and the target. If the privileged channel provides a regular representation of sample difficulty or target structure, certain LUPI analyses obtain faster learning rates than those associated with an otherwise comparable learner.

The frequently cited rate comparison contrasts an order of (O(n^{-1/2})) for a general nonseparable classification problem with an order approaching (O(n^{-1})) under additional regularity supplied by an effective correcting function. These rates describe theoretical regimes rather than a universal empirical law. They rely on assumptions about the relevant function classes, the approximation quality of the privileged representation, and the behavior of the optimization problem.

Privileged information can also be uninformative or misleading. Variables unrelated to the target contribute no transferable structure, while variables that encode artifacts specific to the training environment can distort the learned relation between (x) and (y). Because the privileged channel disappears at prediction time, any useful effect must ultimately be expressible through the ordinary representation.

Relation to adjacent frameworks

LUPI differs from semi-supervised learning, which supplements labeled observations with unlabeled ordinary inputs. It also differs from transfer learning, where information acquired from another task or distribution influences a target problem. In LUPI, the additional information is attached to the labeled training observations and is explicitly unavailable at deployment.

The framework overlaps with multiview learning when one view is restricted to training. Ordinary multiview methods commonly assume that several views remain observable, whereas privileged-view methods preserve only one of them in the prediction function. LUPI also intersects with missing data, but the absence of privileged variables at test time is structural rather than accidental.

Its relation to learning with noisy labels follows from the correcting-function interpretation. Privileged variables can identify observations whose labels or margins are less reliable, allowing the training objective to model heterogeneous difficulty. The final classifier nevertheless remains constrained by the information contained in ordinary inputs.