Semi-supervised learning

Semi-supervised learning is a branch of machine learning that studies inference from a combination of labeled and unlabeled observations. A labeled observation contains an input together with its target value, whereas an unlabeled observation contains only the input. The field lies between supervised learning, in which training data provide target values, and unsupervised learning, in which the training data provide no explicit targets.

Unlabeled data do not determine a predictive rule without additional assumptions about the relation between the input distribution and the target function. Semi-supervised methods therefore incorporate structural assumptions that connect regions of high input density, geometric neighborhoods, or perturbations of an observation with similarities in predicted labels. When these assumptions correspond to the data-generating process, unlabeled observations constrain the set of predictive functions consistent with the labeled sample. When the assumptions fail, the same constraints can reduce predictive accuracy.

Mathematical formulation

Let the labeled sample be

[ \mathcal{D}{L}={(x_i,y_i)}{i=1}^{n_L}, ]

and let the unlabeled sample be

[ \mathcal{D}{U}={x_j}{j=n_L+1}^{n_L+n_U}. ]

Both samples ordinarily occupy the same input space (\mathcal{X}), while each label belongs to an output space (\mathcal{Y}). In classification, (\mathcal{Y}) is a finite set of classes. In regression analysis, it is usually a subset of the real numbers.

A large class of semi-supervised estimators minimizes an objective of the form

[ \mathcal{L}(\theta)

\mathcal{L}_{\mathrm{sup}}(\theta;\mathcal{D}L) + \lambda\mathcal{L}{\mathrm{unsup}}(\theta;\mathcal{D}_U) + \Omega(\theta). ]

The supervised term measures disagreement between predictions and observed labels. The unlabeled term imposes a relationship between predictions and the structure of the input sample, while the regularization term controls properties of the parameterized model. The coefficient (\lambda) determines the relative contribution of the unlabeled objective, rather than converting unlabeled observations into labels by itself.

This formulation includes both inductive learning, which produces a function for future inputs, and transductive inference, which predicts labels for a fixed collection of unlabeled observations. The distinction concerns the inferential target rather than the presence or absence of unlabeled data.

Structural assumptions

The smoothness assumption states that inputs close under an appropriate representation tend to have similar conditional label distributions. Its relevance depends on the geometry induced by the representation, since proximity in raw coordinates does not necessarily correspond to semantic similarity. Regularization based on this assumption penalizes rapid changes in the predictive function across locally dense regions.

The cluster assumption assigns related labels to observations lying within the same high-density component of the input distribution. A corresponding decision boundary passes through a region containing relatively few observations. This principle underlies low-density separation and several forms of entropy minimization, although a density valley does not identify which class belongs on either side without labeled information.

The manifold hypothesis treats high-dimensional observations as concentrated near a lower-dimensional geometric structure. Unlabeled samples then provide information about local neighborhoods and directions of variation along that structure. Graph-based regularization and perturbation-based consistency objectives both approximate aspects of this geometry under different constructions.

These assumptions are logically distinct from the ordinary identically distributed sampling assumption. Labeled and unlabeled observations may share a marginal input distribution while still violating the cluster or smoothness relation required by a particular algorithm. Conversely, distribution shift between the two samples changes the interpretation of the unlabeled objective even when each distribution separately exhibits geometric regularity.

Principal formulations

Generative modeling

A generative approach specifies a joint distribution (p_\theta(x,y)). Labeled observations contribute the joint likelihood (p_\theta(x_i,y_i)), while unlabeled observations contribute the marginal likelihood

[ p_\theta(x_j)=\sum_y p_\theta(x_j,y). ]

Parameter estimation can use the expectation–maximization algorithm, which alternates between estimating latent label distributions and updating model parameters. The unlabeled sample influences the estimated class-conditional input distributions and the class proportions. This influence is statistically informative when the assumed family represents the relevant features of the data, but model misspecification can make the estimated marginal distribution incompatible with accurate classification.

Self-training and pseudo-labels

Self-training begins with a predictor fitted to labeled observations and assigns provisional targets to selected unlabeled observations. These pseudo-labels subsequently enter a supervised-style loss. Modern variants usually retain probabilistic targets or confidence thresholds, thereby representing the provisional status of inferred labels within the optimization objective.

The method creates a feedback process because predictions generated by the current model affect its later parameter updates. Correct provisional labels can extend a decision region through a coherent cluster, whereas systematic errors can reinforce an incorrect boundary. The statistical behavior therefore depends on calibration, class balance, and the relationship between prediction confidence and actual error.

Co-training, introduced by Avrim Blum and Tom Mitchell, uses two feature descriptions whose predictive information is sufficiently distinct under its formal assumptions. Each classifier supplies labeled information for the other classifier through examples on which its own view is informative. Later interpretations replaced strict conditional independence with weaker notions of complementary prediction, but the original analysis depended on a specific multi-view structure.

Graph-based methods

Graph-based semi-supervised learning represents observations as vertices connected by weighted edges. Edge weights encode similarity, and a typical objective penalizes prediction differences between strongly connected vertices. If (W) is the weight matrix and (L=D-W) is the graph Laplacian, a quadratic regularizer takes the form

[ \sum_{i,j}W_{ij}\lVert f(x_i)-f(x_j)\rVert^2

2,\mathrm{tr}(F^\mathsf{T}LF). ]

Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty developed influential label-propagation formulations in which observed labels act as boundary conditions on a graph containing both labeled and unlabeled vertices. Predictions spread through paths of high similarity, making the resulting classification dependent on graph construction as well as on the initial labels.

Graph methods provide a finite-sample representation of local geometric regularity. Their behavior changes when neighborhood relations connect separate classes or fragment a single class into disconnected components. This dependence is a consequence of encoding the learning assumption directly in the graph.

Consistency regularization

Consistency regularization requires predictions to remain stable under transformations that preserve the target. For a stochastic transformation (T), a representative objective compares

[ f_\theta(T(x)) \quad\text{with}\quad f_{\bar{\theta}}(x), ]

where (\bar{\theta}) may denote the same model, a delayed parameter average, or another coupled predictor. The transformation distribution defines which variations the model treats as irrelevant to classification.

In 2008, You Watanabe derived the small-perturbation limit of this objective and expressed its leading term as a Laplacian penalty on the data manifold. The result established a formal correspondence between local prediction agreement and graph-based smoothness when the perturbation covariance represents neighborhood geometry. This treatment also separated invariance induced by the transformation distribution from invariance inferred from input density.

Later neural formulations integrated consistency losses with parameter averaging and pseudo-labeling. Antti Tarvainen and Harri Valpola formulated the mean-teacher model, in which a predictor with exponentially averaged parameters supplies targets for a separately updated student model. The arrangement reduces short-term variation in the target predictions without changing the underlying requirement that perturbations preserve class identity.

Statistical interpretation

Unlabeled inputs provide direct information about the marginal distribution (p(x)), but prediction depends on the conditional distribution (p(y\mid x)). A gain from unlabeled data therefore requires a restriction connecting these distributions. Generative models impose the connection through a joint parametric family, while geometric methods impose it through regularity along estimated neighborhoods. Consistency methods encode the connection through transformations under which the conditional label distribution remains stable.

This distinction explains why increasing (n_U) does not universally reproduce the effect of increasing (n_L). Additional unlabeled observations can estimate input density or manifold geometry with greater precision, but they do not independently resolve the assignment of class identities. A small labeled sample may also omit a class entirely, leaving no supervised basis for attaching that class name to a discovered component.

Theoretical analyses use measures such as sample complexity, margin size, graph connectivity, and model identifiability. These quantities describe different mechanisms and do not combine into a single universal guarantee. Bounds obtained under a cluster assumption characterize low-density separation, whereas bounds for co-training depend on relationships between feature views. Generative analyses instead depend on whether distinct parameter values induce distinguishable observed-data distributions.

Evaluation and failure modes

Semi-supervised evaluation separates labeled training data from labeled validation and test data. Unlabeled test inputs are not equivalent to an unlabeled training sample unless the task is explicitly transductive, because access to the evaluation inputs changes the inferential setting. Comparisons with a supervised baseline identify whether the unlabeled objective contributes information beyond the architecture and labeled-data regularization.

Distribution mismatch is a principal source of failure. An unlabeled collection may contain classes absent from the labeled sample, or its class proportions may differ substantially from those represented by the labels. A confidence-based objective can then assign unsupported inputs to familiar classes and amplify the resulting error.

Incorrect invariance produces another failure mechanism. A transformation that changes the true label contradicts the consistency objective, even when the transformed input appears visually or statistically similar. Graph methods exhibit the corresponding problem when an edge joins observations from different classes. In each case, the algorithm enforces a mathematically coherent relation that does not match the target-generating process.

Performance also depends on the labeled sample’s placement within the input geometry. Labels concentrated in one region may leave other connected components weakly identified, while class imbalance can make confident majority-class predictions dominate pseudo-label selection. These effects arise from the interaction between supervised information and the structure assigned to unlabeled observations.