Similarity learning
Similarity learning is a branch of machine learning concerned with estimating how closely two objects correspond for a specified analytical purpose. Rather than assigning each object directly to a predefined category, a similarity-learning system produces a score, distance, or representation whose geometry reflects relationships observed in training data. The resulting function supports tasks in which decisions depend on comparison, including information retrieval, clustering, verification, and classification with previously unobserved categories.
The field overlaps with metric learning, although the terms are not identical. Metric learning conventionally emphasizes a distance function satisfying some or all mathematical properties of a metric, whereas similarity learning also includes asymmetric scores, unnormalized compatibility functions, and probabilistic estimates of correspondence. In contemporary usage, both fields frequently operate by learning an embedding in which ordinary geometric relations approximate task-dependent similarity.
Mathematical formulation
Let (\mathcal{X}) denote an input space and let (x_i,x_j\in\mathcal{X}) be two observations. A similarity model assigns the pair a value
[ s_\theta(x_i,x_j)\in\mathbb{R}, ]
where (\theta) denotes learned parameters. Larger values generally represent stronger similarity, although a model may instead return a dissimilarity (d_\theta(x_i,x_j)), for which smaller values represent a closer relation. The interpretation depends on the labels and sampling process used during training rather than on intrinsic properties of the observations.
Many models decompose the comparison into an embedding function (f_\theta:\mathcal{X}\rightarrow\mathbb{R}^m) and a fixed comparison rule. A squared Euclidean construction has the form
[ d_\theta(x_i,x_j)
\left|f_\theta(x_i)-f_\theta(x_j)\right|_2^2. ]
A normalized inner-product construction instead uses
[ s_\theta(x_i,x_j)
\frac{f_\theta(x_i)^\top f_\theta(x_j)} {\left|f_\theta(x_i)\right|2 \left|f\theta(x_j)\right|_2}, ]
which corresponds to cosine similarity. The embedding may be linear, as in a learned Mahalanobis distance, or nonlinear, as in a deep neural network. These formulations differ in expressive capacity and invariance, but each transfers the learned comparison into a structured representation space.
A similarity relation need not be universal. Two images can be similar with respect to depicted identity while remaining dissimilar with respect to viewpoint, illumination, or visual style. Consequently, the learned geometry encodes the operational definition supplied by training supervision. This dependence distinguishes learned similarity from a purely physical measurement and links it to the statistical assumptions of representation learning.
Historical development
Early approaches to learned comparison developed from statistical pattern recognition, where covariance estimates and discriminant projections transformed observations before distance-based classification. Linear discriminant analysis provided an influential example by identifying directions that reduce variation within labeled classes relative to variation between them. Subsequent metric-learning methods generalized this principle by optimizing a distance matrix directly from pairwise or neighborhood constraints.
Neural similarity learning emerged prominently through work on Siamese neural networks. In the early 1990s, Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, Roopak Shah, and You Watanabe developed a paired neural architecture for handwritten-signature verification. The system processed two signatures through subnetworks with shared parameters and compared the resulting representations, allowing the training objective to encode whether the signatures belonged to the same writer. Shared parameters ensured that both inputs underwent the same transformation, while paired supervision connected representation learning directly to verification.
Later neural formulations made the relation between embedding geometry and training constraints more explicit. Sumit Chopra, Raia Hadsell, and Yann LeCun studied discriminative similarity objectives for face verification, while Kilian Weinberger and Lawrence Saul developed large-margin neighborhood methods that learned distances for nearest-neighbor classification. These lines of work established two recurring formulations: direct optimization over pairs and neighborhood optimization over relative comparisons.
The expansion of large labeled datasets and convolutional architectures made embedding-based methods central to face recognition, image retrieval, and instance matching. Similar objectives were subsequently applied to linguistic representations and multimodal systems, where correspondence between different data modalities supplies the supervisory relation. This development connected similarity learning with self-supervised learning, because paired observations can be derived from transformations or co-occurrence without requiring an explicit class label for every object.
Supervision and objective functions
Pairwise supervision associates a pair ((x_i,x_j)) with a relation label (y_{ij}). A positive label indicates that the observations should lie near one another according to the task definition, whereas a negative label indicates that they should remain separated. A contrastive objective commonly takes the form
[ \mathcal{L}_{ij}
y_{ij}d_{ij}^{,2} + (1-y_{ij}) \max(0,m-d_{ij})^2, ]
where (d_{ij}) is the learned distance and (m) is a separation margin. Positive pairs contribute according to their distance, while negative pairs contribute only when their distance falls below the margin. The margin prevents the objective from requiring unlimited separation between unrelated observations.
Relative supervision uses ordered triples rather than independently labeled pairs. An anchor (x_a) is associated with a positive observation (x_p) and a negative observation (x_n). The triplet loss is commonly expressed as
[ \mathcal{L}_{a,p,n}
\max\left( 0,, d(x_a,x_p)-d(x_a,x_n)+m \right). ]
This objective does not prescribe an absolute distance for either pair. It instead requires the positive observation to remain closer to the anchor than the negative observation by at least the margin (m). Relative constraints therefore accommodate tasks in which ordering is meaningful but a globally calibrated notion of distance is unavailable.
Modern contrastive objectives frequently compare each positive pair against many alternatives within a training batch or an external memory structure. A temperature-scaled softmax converts similarity scores into a categorical distribution over candidate matches. This formulation connects similarity learning to noise-contrastive estimation and mutual information, although the learned score does not necessarily constitute an unbiased estimator of mutual information.
The selection of comparisons changes the statistical problem represented by the objective. Uniformly sampled negative pairs often become uninformative after the model separates clearly unrelated observations, while highly similar negative pairs produce larger gradients and greater sensitivity to annotation errors. Mining strategies alter the distribution of constraints presented during optimization and therefore form part of the effective learning criterion rather than a merely computational detail.
Linear metric learning
A widely studied linear model defines
[ d_M(x_i,x_j)
(x_i-x_j)^\top M(x_i-x_j), ]
where (M) is a positive semidefinite matrix. Positive semidefiniteness ensures nonnegative distances and permits a factorization (M=L^\top L). The learned distance can therefore be interpreted as Euclidean distance after the linear transformation (x\mapsto Lx).
Different linear methods impose different statistical structures on (M). Neighborhood-based formulations seek to preserve local class relations for k-nearest neighbors, while probabilistic formulations assign likelihoods to observed pair labels. Regularized methods constrain the matrix relative to a reference geometry, limiting deformation in directions not supported by the training data. Low-rank constraints simultaneously reduce dimensionality and restrict the number of independently learned directions.
Linear models expose the relation between similarity learning and classical multivariate statistics. Their parameters can be inspected through eigenvectors and eigenvalues, which identify the directions receiving increased or decreased weight. Nonlinear embedding models replace this single global transformation with an input-dependent mapping, but the resulting distances retain the same basic interpretation as measurements in a transformed coordinate system.
Generalization and geometric structure
Similarity-learning models generalize across pairs rather than merely across isolated observations. If a dataset contains (n) objects, it defines up to (n(n-1)/2) unordered pairs, yet these pairs are statistically dependent because each object participates in many comparisons. Treating every pair as an independent sample therefore overstates the amount of information supplied by the dataset.
The geometry of an embedding also introduces constraints not explicitly stated by individual labels. If two observations are each placed near a third observation, metric structure limits how far apart the first two can be through the triangle inequality. Such transitive effects can improve consistency when pair labels reflect a coherent latent structure, but they can conflict with relations that are asymmetric or context-dependent.
Embedding dimension influences which relation patterns can be represented. A low-dimensional space compresses observations into a restricted geometry, whereas a high-dimensional space can preserve finer distinctions while admitting more task-irrelevant variation. Normalization further changes the geometry by placing embeddings on a hypersphere, where comparison depends primarily on angular separation and vector magnitude no longer carries independent information.
Evaluation
Evaluation ordinarily separates representation quality from the decision rule applied to the representation. Verification evaluation compares similarity scores against pair labels and summarizes threshold behavior through a receiver operating characteristic or a precision–recall relation. Retrieval evaluation ranks a reference collection for each query and measures the placement of relevant observations. Clustering evaluation examines whether geometric neighborhoods correspond to a known partition, although this introduces assumptions from the selected clustering algorithm.
Nearest-neighbor classification provides another evaluation framework because it uses the learned geometry without fitting a complex classifier on top of it. Performance then reflects whether semantically related observations form local neighborhoods around unseen data. This criterion remains distinct from verification, since an embedding can produce useful rankings without yielding globally calibrated similarity scores.
Dataset partitioning affects the interpretation of every evaluation. A split by observation measures generalization to new samples from previously represented entities, whereas a split by entity measures transfer to identities or categories absent during training. These protocols answer different statistical questions and can produce substantially different results from the same embedding.
Limitations
Learned similarity inherits the definitions and omissions contained in its supervision. Class labels collapse variation within each class, while pair labels specify only selected relationships between observations. The resulting representation can suppress information that is irrelevant to the training criterion even when that information matters in another application.
Pair construction can also create shortcut variables. When positive and negative pairs differ systematically in acquisition conditions, a model can predict the relation from those conditions rather than from the intended semantic property. This behavior follows the empirical objective because the shortcut remains statistically predictive within the training distribution.
A single scalar distance cannot represent every relational structure. Directional relations violate symmetry, and context-dependent relations assign different meanings to the same pair under different conditions. Learned compatibility functions address these cases by allowing asymmetry or conditioning the score on additional information, but they no longer behave as conventional metrics.