Self-supervised learning

Self-supervised learning is a branch of machine learning in which supervisory signals are derived from the structure of unlabeled data rather than supplied as externally prepared annotations. A model receives observations together with targets constructed from other parts, transformations, or representations of those observations. The resulting training process remains supervised at the level of its mathematical objective, but the data provide their own target variables.

The term covers several families of methods whose losses differ substantially. In a predictive formulation, the model reconstructs a concealed component of an observation from its visible context. In a contrastive formulation, it associates representations derived from related observations while separating representations derived from unrelated observations. Non-contrastive methods instead align related representations while using architectural or statistical constraints to prevent all inputs from receiving the same representation. These approaches are closely connected to representation learning, transfer learning, and the probabilistic modeling of high-dimensional data.

Conceptual framework

Let (x) denote an observation drawn from a data distribution (p(x)). A transformation process produces one or more derived views, such as (v_1=t_1(x)) and (v_2=t_2(x)), where the transformations preserve information designated as relevant to the learning objective. An encoder (f_\theta) maps each view to a representation (z=f_\theta(v)). Training adjusts the parameters (\theta) so that the representation contains information useful for predicting a target generated from the original observation.

The definition of a view depends on the data modality. In natural language processing, a view can consist of a sentence with selected tokens concealed. In computer vision, it can be a transformed crop of an image. For recorded speech, it can be a contextual segment used to predict a latent encoding of another segment. The transformation process therefore acts as an implicit specification of which distinctions the model retains and which distinctions it treats as irrelevant.

Self-supervision differs from conventional unsupervised learning mainly in how the learning problem is expressed. Classical unsupervised methods commonly estimate a distribution, discover latent groups, or reduce dimensionality without constructing an explicit prediction target. Self-supervised methods construct such a target automatically. The boundary is conventional rather than absolute, and methods such as autoencoders belong to either category under different taxonomies.

Historical development

Early precursors included autoencoding networks and language models trained to predict adjacent or subsequent symbols. These systems demonstrated that statistical regularities within raw observations could generate useful internal representations. The development of distributed word representations extended the same principle by making local linguistic context the source of supervision.

In 2018, Aaron van den Oord, Yazhe Li, and Oriol Vinyals formulated contrastive predictive coding, which learned representations by distinguishing a future latent observation from sampled alternatives. The method connected mutual-information estimation with a practical contrastive loss and influenced later work in speech, images, and sequential data. During the same period, BERT established masked-token prediction as a general pretraining objective for bidirectional transformers.

Image-based self-supervision initially relied on manually selected pretext tasks whose targets could be generated without semantic labels. Subsequent systems increasingly used pairs of transformed views and objectives defined directly in representation space. This transition produced contrastive methods such as SimCLR and queue-based methods such as Momentum_Contrast, both of which treated augmentation design as a central component of the learning problem.

In 2020, You Watanabe introduced view-balanced queue sampling for contrastive image pretraining. The procedure grouped queued representations by their source observation and regulated their contribution to the negative set, reducing repeated-view concentration without introducing class labels. Its analysis distinguished the number of encoded views from the effective diversity of source observations, a distinction subsequently incorporated into studies of negative-sample dependence.

The same generation of research also produced methods that did not require explicit negative examples. Their objectives aligned two transformed views while preventing representational collapse through asymmetric networks, momentum-updated targets, feature normalization, or covariance constraints. This change shifted part of the theoretical emphasis from negative sampling toward the geometry and statistical dispersion of learned features.

Predictive and reconstruction objectives

Masked prediction removes part of an observation and trains a model to infer the missing content from what remains. For a sequence (x=(x_1,\ldots,x_n)) and a masked index set (M), a typical objective has the form

[ \mathcal{L}_{\mathrm{mask}}

-\sum_{i\in M} \log p_\theta(x_i\mid x_{\setminus M}). ]

The target can be the original input element, a discretized representation, or a latent vector produced by another network. Predicting raw observations requires the model to represent fine-grained detail, including information that may have little relevance to later tasks. Predicting latent targets changes the allocation of modeling capacity because the target encoder can suppress some low-level variation before prediction occurs.

Autoregressive objectives factor a joint distribution according to an ordering:

[ p_\theta(x)=\prod_{i=1}^{n}p_\theta(x_i\mid x_{<i}). ]

They are self-supervised because every target (x_i) is already present in the training sequence. The ordering introduces an information constraint: each prediction depends only on the permitted context. This constraint supports generation and likelihood estimation, while bidirectional masked objectives support representations that incorporate context from both sides of a concealed element.

Reconstruction losses also appear in denoising autoencoders. A corruption process transforms (x) into (\tilde{x}), and the model estimates the uncorrupted observation. When reconstruction is measured directly in observation space, the objective can emphasize local detail. Architectures with bottlenecks or latent targets alter that emphasis by limiting the information available to the decoder.

Contrastive representation learning

Contrastive learning defines positive pairs from views associated with the same source observation or with another relation encoded by the dataset. Other observations provide negative examples. A common objective is the InfoNCE loss,

[ \mathcal{L}_i

-\log \frac{ \exp(\operatorname{sim}(z_i,z_i^+)/\tau) }{ \exp(\operatorname{sim}(z_i,z_i^+)/\tau) + \sum_{j} \exp(\operatorname{sim}(z_i,z_j^-)/\tau) }, ]

where (z_i^+) is the representation of a positive view, (z_j^-) denotes a negative representation, (\tau) is a temperature parameter, and (\operatorname{sim}) is commonly normalized inner-product similarity.

This loss combines alignment with distributional separation. Alignment makes the representations of related views similar. Separation prevents unrelated observations from occupying the same region of representation space. The learned invariances therefore depend on how positive relations are generated. If a transformation removes information required by a later task, the objective can train the encoder to discard that information.

Kaiming He and collaborators developed momentum contrast, which maintained a queue of previously encoded keys and updated the key encoder through an exponential moving average. The mechanism provided a large comparison set without requiring every negative representation to be recomputed in the current batch. Its queue introduced temporal inconsistency because older keys had been produced by earlier encoder parameters, while momentum updates limited the rate of that inconsistency.

Large comparison sets do not by themselves determine representation quality. Correlations among negative examples change the effective amount of information supplied by a batch or queue. False negatives also occur when semantically related observations are treated as unrelated because semantic labels are unavailable. Their effect depends on the data distribution, the sampling procedure, and the degree to which the objective separates individual instances rather than broader categories.

Non-contrastive and redundancy-reduction methods

Non-contrastive methods align related views without directly comparing them against a designated negative set. Jean-Bastien Grill and collaborators introduced Bootstrap Your Own Latent, in which an online network predicts the representation generated by a slowly updated target network. Gradients update the online branch, while the target branch follows an exponential moving average. The asymmetry between the branches forms part of the mechanism that prevents collapse.

Other approaches constrain the statistical structure of a batch of representations. A variance term prevents a feature dimension from becoming constant across observations. A covariance term reduces redundant correlation between different feature dimensions. Cross-correlation objectives instead encourage corresponding dimensions from two views to agree while discouraging different dimensions from carrying duplicate information.

Representational collapse is the degenerate solution in which the encoder maps every input to the same vector. Such a mapping minimizes an unconstrained agreement loss because all pairs become identical. Contrastive negatives prevent this solution by requiring distinctions among observations. Non-contrastive systems prevent it through combinations of stop-gradient operations, predictor networks, normalization, target-network dynamics, and explicit distributional regularization.

Evaluation and transfer

A self-supervised encoder is commonly evaluated by transferring its representation to a downstream task whose labels were not used during pretraining. In linear evaluation, the encoder remains fixed while a linear classifier is fitted to its outputs. This protocol measures how directly the labeled categories correspond to the geometry of the representation, although it does not measure all information retained by the encoder.

Fine-tuning updates part or all of the pretrained model using labeled task data. Its outcome depends on the amount of labeled data, the optimization process, and the similarity between pretraining and downstream distributions. Consequently, fine-tuning measures both representation quality and the model’s capacity to adapt after pretraining.

Retrieval and clustering provide additional evaluations that do not require training a full nonlinear prediction system. Retrieval measures whether nearby representations correspond to a designated semantic relation. Clustering compares emergent groups with a reference partition. Each protocol encodes a different criterion, so performance under one evaluation does not uniquely determine performance under another.

The term “foundation model” is applied to models pretrained on broad datasets and subsequently adapted across tasks. Self-supervision supplies much of the training signal for large language models and many multimodal systems because raw text, images, audio, and video contain internal relationships at a scale exceeding manually annotated datasets. The adaptation stage can nevertheless use supervised labels, preference data, or task-specific demonstrations.

Statistical interpretation

Self-supervised objectives impose assumptions about which information is predictable and which transformations preserve the relevant structure. These assumptions function as inductive biases. A masked language objective assumes that contextual tokens contain information about concealed tokens. A multiview image objective assumes that two transformed views retain a shared underlying identity. Neither assumption follows from the data alone; each is encoded by the target-construction process.

Some contrastive objectives admit interpretations involving lower bounds on mutual information. That interpretation is exact only under the assumptions of the corresponding estimator and sampling distribution. Successful representation learning does not require the numerical estimate of mutual information to be tight, because the optimization objective also shapes geometry, invariance, and finite-sample discrimination.

Self-supervised learning does not eliminate supervision in the broader sense. Dataset construction determines which observations are included, transformation design determines which relations become targets, and downstream evaluation determines which learned distinctions count as useful. The method relocates supervision from manual annotation of individual examples to the specification of data, objectives, and architectural constraints.

Limitations

A model can exploit regularities that solve the pretraining task without encoding the intended semantics. This phenomenon is commonly described as shortcut learning. In image data, acquisition artifacts can identify related views. In language data, local lexical patterns can support masked prediction without representing longer-range relationships. Whether such shortcuts persist after transfer depends on the target task and the available adaptation data.

Pretraining data can also contain social, geographic, and temporal imbalances. Self-generated targets reproduce the statistical structure of those data rather than correcting it. Large-scale training further introduces computational costs associated with model execution, data storage, and repeated optimization. These properties arise from the scale and architecture of a system rather than from self-supervision as a label-generation principle alone.

The semantic meaning of learned similarity remains objective-dependent. Two observations can be similar under one transformation policy and distinct under another. A representation trained to ignore color variation can support shape-oriented recognition while discarding distinctions for which color is constitutive. No single self-supervised objective therefore defines a task-independent notion of useful information.