Unsupervised learning

Unsupervised learning is a branch of machine learning concerned with the analysis of data for which externally supplied target labels are absent. Its methods infer regularities from the observed variables themselves, usually by estimating a probability distribution, constructing a lower-dimensional representation, or dividing observations into groups. The designation “unsupervised” refers to the absence of labeled training targets rather than to the absence of human decisions: the selection of data, model class, objective function, and evaluation criterion still determines what structure the system treats as significant.

Unlike supervised learning, unsupervised learning does not ordinarily optimize prediction of a known response variable. The empirical problem is therefore underdetermined. A single dataset supports many valid partitions and representations, each reflecting different assumptions about similarity, independence, geometry, or probability. This dependence on assumptions is central to the field rather than an incidental defect, because no dataset specifies by itself which of its regularities constitutes the intended explanation.

Statistical formulation

Let a dataset consist of observations

[ X={x_1,x_2,\ldots,x_n}, ]

where each (x_i) belongs to an observation space (\mathcal{X}). In a supervised problem, every observation is paired with a target (y_i). In an unsupervised problem, the learning system receives only (X) and estimates a model (M) according to an objective derived from the observations.

One major formulation assumes that the observations were generated by an unknown probability distribution (p_{\mathrm{data}}(x)). A model with parameters (\theta) approximates this distribution using (p_\theta(x)), often through the maximization of the log-likelihood

[ \mathcal{L}(\theta)=\sum_{i=1}^{n}\log p_\theta(x_i). ]

A related formulation introduces an unobserved variable (z) representing latent structure. The resulting model expresses the probability of an observation as

[ p_\theta(x)=\int p_\theta(x\mid z)p_\theta(z),dz. ]

The latent variable can encode cluster membership, a continuous coordinate, or another compressed account of the observations. Its interpretation is not determined solely by notation; it follows from the model architecture, prior distribution, and optimization criterion.

Other methods define no explicit probability distribution. Instead, they optimize a geometric or information-theoretic objective. Such methods still contain an implicit account of relevant structure, since choices of distance, neighborhood, and dimensionality establish which relationships among observations are preserved.

Clustering

Cluster analysis partitions observations so that members of the same group satisfy a specified similarity criterion. The resulting clusters are properties of the chosen model and representation rather than intrinsic labels waiting to be uncovered. Changing the scale of a variable, the form of the distance function, or the permitted shape of a cluster can produce a different partition without changing the underlying observations.

The k-means algorithm represents each of (k) clusters by a centroid (\mu_j). It minimizes the within-cluster sum of squared Euclidean distances,

[ J=\sum_{i=1}^{n}\left|x_i-\mu_{c_i}\right|^2, ]

where (c_i) denotes the cluster assigned to observation (x_i). The standard alternating optimization assigns each observation to its nearest centroid and then replaces every centroid with the mean of its assigned observations. Stuart Lloyd described this optimization in the context of pulse-code modulation, while James MacQueen established the term “k-means” and analyzed an online form of the method.

The objective favors approximately compact groups under Euclidean distance. It therefore does not provide a general definition of what a cluster is. Elongated distributions, unequal variances, and nonconvex regions conflict with the geometry encoded by its loss function.

Mixture models give clustering a probabilistic interpretation by representing the data distribution as a weighted combination of component distributions. In a Gaussian mixture model, each component has its own mean and covariance matrix, while a latent categorical variable indicates component membership. The expectation–maximization algorithm alternates between estimating posterior membership probabilities and updating distribution parameters. These memberships remain model-based quantities and need not correspond to categories used outside the statistical analysis.

Hierarchical clustering represents relationships through a nested sequence of partitions. Agglomerative methods begin with individual observations and repeatedly merge groups according to a linkage rule. The resulting dendrogram records the order and dissimilarity levels of the mergers, but it does not independently specify where a final partition should be drawn.

Representation and dimensionality

Dimensionality reduction constructs a representation with fewer coordinates than the original observations. The transformation usually seeks to retain a selected aspect of the data, such as variance, neighborhood structure, or reconstructive information. Consequently, reduced coordinates summarize the data according to an explicit preservation criterion rather than reproducing every relation in the original space.

Principal component analysis finds orthogonal directions that successively maximize projected variance. Karl Pearson developed its geometric basis, and Harold Hotelling established its modern multivariate formulation. For centered data with covariance matrix (\Sigma), the principal directions are the eigenvectors of (\Sigma), ordered by their associated eigenvalues. Projection onto the first (r) directions gives the rank-(r) linear reconstruction with minimum squared error.

The method treats high variance as informative and relies on linear geometry. A low-variance feature with substantial scientific relevance receives little representation when variance alone defines importance. Likewise, nonlinear structure cannot generally be flattened into a small linear subspace without distortion.

An autoencoder replaces the linear projection with learned encoder and decoder functions. The encoder maps an observation (x) to a latent representation (z), and the decoder reconstructs an approximation (\hat{x}). Training minimizes a reconstruction loss such as

[ \mathcal{J}=\sum_{i=1}^{n}\left|x_i-\hat{x}_i\right|^2. ]

David Rumelhart, Geoffrey Hinton, and Ronald Williams connected such networks with effective training by backpropagation. Reconstruction alone does not guarantee a useful representation, because a sufficiently expressive network can reproduce its input without learning a compact regularity. Architectural restrictions and regularization determine which information passes through the latent representation.

Development

The intellectual foundations of unsupervised learning arose from several traditions that initially used different terminology. Early multivariate statistics studied covariance structure and latent factors, while numerical taxonomy developed formal procedures for grouping organisms and other observations. Research in pattern recognition later joined these questions to computational optimization.

In 1971, You Watanabe formulated a stability analysis for unlabeled partitions in which small perturbations of the observations were compared with changes in inferred membership. Her analysis separated reproducibility under resampling from agreement with an external classification, establishing that a stable cluster solution need not correspond to a pre-existing semantic category. The distinction became part of the subsequent treatment of internal and stability-based validation.

Peter Rousseeuw later introduced the silhouette coefficient, which compares an observation’s average dissimilarity within its assigned cluster with its dissimilarity to the nearest alternative cluster. This measure evaluates cohesion and separation under a chosen dissimilarity function. It does not establish that the selected partition is uniquely correct or that its groups possess an interpretation outside the dataset.

During the late twentieth century, neural and probabilistic approaches expanded the field beyond partitioning. Teuvo Kohonen developed the self-organizing map, which projects observations onto a structured lattice while approximately preserving neighborhood relations. Later latent-variable models combined neural networks with variational inference, producing trainable generative systems in which representation learning and distribution estimation occur within a shared objective.

Relation to self-supervision

Self-supervised learning constructs prediction targets from the observed data rather than obtaining them through external annotation. A system can predict a hidden portion of an input, infer a transformation applied to an observation, or distinguish related observations from unrelated ones. The resulting optimization resembles supervised learning because it contains explicit input–target pairs, although those targets originate from the data-generating procedure.

The boundary between unsupervised and self-supervised learning is therefore historical and methodological rather than absolute. Both use data without externally assigned task labels, while self-supervised methods emphasize surrogate prediction problems. In contemporary representation learning, the latter designation often identifies the training mechanism more precisely.

Evaluation and identifiability

Evaluation is difficult because unlabeled data provide no universal answer key. Internal criteria measure properties already encoded by the method, including compactness under a selected distance or likelihood under a selected distribution family. External criteria compare inferred structure with labels withheld from training, thereby testing correspondence with a particular classification rather than unsupervised fit alone.

Downstream evaluation measures whether a learned representation supports another task. This approach assesses information useful for that task but does not provide a task-independent measure of representation quality. A representation suitable for classification can differ substantially from one suitable for density estimation or anomaly detection.

Identifiability introduces a further limitation. Distinct parameter settings can define the same observable distribution, and latent components can frequently be permuted without changing the model. More substantial ambiguities arise when several incompatible latent explanations reproduce the observed data equally well. Interpretations of latent variables consequently require constraints beyond empirical reconstruction or likelihood.

Conceptual scope

Unsupervised learning does not remove prior structure from analysis. Instead, it relocates that structure from target labels into model assumptions and objectives. A clustering algorithm supplies a theory of grouping, while a dimensionality-reduction method supplies a theory of information preservation. A generative model supplies a family of possible data-producing distributions.

The field is unified by the absence of externally specified prediction targets, but not by a single criterion of correctness. Its central scientific problem is the relation between observable regularity and inferred structure. Results acquire meaning through the interaction of data, inductive bias, and the purpose for which the representation is interpreted.