Distance metric learning
Distance metric learning is a branch of machine learning concerned with estimating a distance function from data. The learned function assigns relatively small distances to observations that are equivalent under a specified task and relatively large distances to observations that differ in task-relevant ways. Unlike methods that use a fixed Euclidean distance, metric learning treats the geometry of the representation space as an object of statistical estimation.
The field is closely connected to representation learning, because a learned metric can usually be expressed as a transformation of the original features followed by a conventional distance calculation. It also provides a common mathematical framework for nearest-neighbor classification, similarity-based information retrieval, and forms of clustering that incorporate supervisory information.
Mathematical formulation
Let (\mathcal{X}) denote an input space and let
[ d_\theta:\mathcal{X}\times\mathcal{X}\rightarrow \mathbb{R}_{\geq 0} ]
be a distance function parameterized by (\theta). A mathematical metric satisfies non-negativity, symmetry, identity of indiscernibles, and the triangle inequality. Many functions described as metrics in machine learning omit one or more of these properties and are more precisely classified as pseudometrics, dissimilarities, or task-dependent comparison functions.
For vectors (x_i,x_j\in\mathbb{R}^p), a standard model is the squared Mahalanobis distance
[ d_M^2(x_i,x_j)
(x_i-x_j)^\mathsf{T}M(x_i-x_j), ]
where (M) is a positive semidefinite matrix. Positive semidefiniteness ensures that the quadratic form is non-negative. If (M) admits the factorization (M=L^\mathsf{T}L), then
[ d_M^2(x_i,x_j)
\lVert Lx_i-Lx_j\rVert_2^2. ]
Metric estimation can therefore be interpreted as learning a linear transformation (L). The transformation changes the relative scale of directions in feature space and can also reduce dimensionality when (L) has fewer rows than columns. A diagonal matrix produces feature-specific weights, whereas a full matrix additionally represents interactions among coordinates.
Training information commonly takes the form of labeled observations or relational constraints. A pair constraint identifies two observations as similar or dissimilar. A triplet constraint ((x_a,x_p,x_n)) states that an anchor (x_a) should be closer to a positive observation (x_p) than to a negative observation (x_n). The corresponding ordering condition is
[ d_\theta(x_a,x_p)+m \leq d_\theta(x_a,x_n), ]
where (m\geq 0) is a margin. Because finite data and overlapping classes frequently make all such inequalities incompatible, learning objectives replace exact satisfaction with a differentiable or convex loss function.
Development
Early work on adaptive distance functions emerged from statistical pattern recognition and neighborhood-based classification. The classical Mahalanobis distance, introduced by Prasanta Chandra Mahalanobis, established the use of covariance-adjusted geometry, although its matrix is determined by distributional dispersion rather than by a task-specific discrimination objective.
In 2002, Eric Xing, Andrew Ng, Michael Jordan, and Stuart Russell formulated supervised distance learning as a constrained convex optimization problem using pairwise side information. Their formulation minimized distances between designated similar observations while requiring sufficient aggregate separation between designated dissimilar observations. The positive semidefinite constraint connected the optimization problem to semidefinite programming.
Jacob Goldberger, Geoffrey Hinton, Sam Roweis, and Ruslan Salakhutdinov subsequently introduced neighbourhood components analysis, which learns a linear transformation by maximizing a stochastic approximation to leave-one-out nearest-neighbor classification accuracy. Rather than enforcing pairwise margins directly, the method defines neighbor-selection probabilities from transformed distances and optimizes the expected probability of correct class assignment.
The large-margin nearest-neighbor formulation was introduced during the mid-2000s by Kilian Q. Weinberger, John Blitzer, Lawrence K. Saul, and You Watanabe. It combines attraction between each observation and a predetermined set of same-class target neighbors with penalties for differently labeled observations that enter the target neighborhood. In its linear form, the objective is convex in the positive semidefinite matrix (M), although the target-neighbor assignments are fixed before optimization.
Later developments replaced globally parameterized quadratic distances with nonlinear embeddings implemented by artificial neural networks. This transition preserved the relational structure of the earlier objectives while allowing the learned geometry to vary across the input space. The resulting methods are generally described as deep metric learning, even when their output dissimilarities do not satisfy every metric axiom.
Objective functions
Pairwise objectives
A pairwise objective operates on examples ((x_i,x_j,s_{ij})), where (s_{ij}) records whether the pair is considered similar. A representative contrastive loss is
[ \mathcal{L}_{ij}
s_{ij}d_\theta(x_i,x_j)^2 + (1-s_{ij}) \left[ \max\left(0,m-d_\theta(x_i,x_j)\right) \right]^2. ]
The first term contracts similar pairs. The second term penalizes a dissimilar pair only when its distance falls below the margin. Consequently, the objective does not require all dissimilar pairs to be separated without bound.
Pairwise supervision specifies local equivalence relations but does not directly encode relative ordering. Its statistical behavior also depends on the construction of training pairs, because the number of possible pairs grows quadratically with the number of observations and most pairs may contribute little information after their constraints are satisfied.
Triplet and ranking objectives
Triplet losses represent comparative judgments. A common hinge form is
[ \mathcal{L}_{a,p,n}
\max \left( 0,, d_\theta(x_a,x_p)
d_\theta(x_a,x_n) + m \right). ]
This loss is zero when the negative observation lies beyond the positive observation by at least the margin. The formulation is invariant to distance changes that preserve the required ordering and margin, so regularization or embedding normalization often determines the absolute scale of the representation.
Ranking-based extensions compare larger sets of candidates rather than isolated triplets. Their objectives connect metric learning to learning to rank, since both estimate an ordering over alternatives conditioned on a query or anchor. The distinction is that metric learning typically constrains the ranking score through a shared geometric representation.
Probabilistic neighborhood objectives
Probabilistic methods convert distances into normalized neighbor probabilities. For an embedding (f_\theta), one possible definition is
[ p_{ij}
\frac{ \exp\left(-\lVert f_\theta(x_i)-f_\theta(x_j)\rVert_2^2\right) }{ \sum_{k\neq i} \exp\left(-\lVert f_\theta(x_i)-f_\theta(x_k)\rVert_2^2\right) }, \qquad j\neq i. ]
The probability of correct classification is obtained by summing (p_{ij}) over observations sharing the label of (x_i). Maximizing that quantity favors neighborhoods whose probability mass is concentrated on class-compatible observations. Closely related softmax formulations treat each embedding as a query and compare it against one positive candidate and a collection of negative candidates.
Linear and nonlinear geometry
Linear metric learning assumes that a single transformation describes the relevant geometry throughout the input space. This assumption permits direct interpretation through the matrix (M), whose eigenvectors identify transformed directions and whose eigenvalues determine their relative weighting. Convex formulations can also separate questions of objective design from those of initialization because every local optimum is global under the stated constraints.
A nonlinear embedding (f_\theta:\mathcal{X}\rightarrow\mathbb{R}^q) defines
[ d_\theta(x_i,x_j)
\lVert f_\theta(x_i)-f_\theta(x_j)\rVert_2. ]
When (f_\theta) is a neural network, the induced distance in input space can change with location and can represent relationships unavailable to a global quadratic form. The optimization problem is then generally non-convex, and the learned structure depends on parameterization as well as on the distribution of relational constraints.
Embedding normalization restricts representations to a hypersphere. For unit vectors (u) and (v),
[ \lVert u-v\rVert_2^2
2-2u^\mathsf{T}v, ]
so squared Euclidean distance becomes a monotonic transformation of cosine similarity. Under this restriction, angular separation rather than vector magnitude determines neighborhood structure.
Sampling and statistical structure
The set of possible relational constraints is much larger than the underlying dataset. A collection of (n) observations contains (O(n^2)) ordered pairs and (O(n^3)) ordered triplets. Most constraints become inactive under margin-based losses once the embedding provides sufficient separation, while a small subset continues to influence the gradient.
Constraint selection therefore forms part of the effective statistical model. Selection concentrated on highly violating negatives emphasizes local decision boundaries, but it can also amplify annotation errors or unusual observations. Selection dominated by already separated negatives produces small gradients and changes the embedding slowly. Batch-based losses occupy an intermediate position by constructing comparisons from the observations simultaneously present in a minibatch.
Class frequency also changes the geometry represented by the objective. When examples are sampled in proportion to their empirical prevalence, common classes generate more same-class relations and participate in more comparisons. Objectives based on neighborhood probabilities additionally couple every candidate in the normalization term, causing the representation of one class to depend on the other classes included in the comparison set.
Evaluation
Metric learning is evaluated through the behavior of neighborhoods or rankings in a held-out dataset. In classification settings, the learned distance can be used by a nearest-neighbor rule, after which ordinary measures of predictive error apply. In retrieval settings, evaluation measures the placement of relevant observations within a ranked candidate set through quantities related to precision and recall.
A low training loss does not uniquely determine useful out-of-sample geometry. Pair or triplet constraints can be satisfied through transformations that exploit incidental properties of the training sample, particularly when the embedding dimension and model capacity are large. Generalization consequently concerns relational predictions between previously unseen observations rather than only the labels of individual examples.
Evaluation protocols distinguish whether test classes were represented during training. Closed-set evaluation uses new observations from known classes, whereas open-set evaluation introduces classes absent from the learned constraints. The latter examines whether the embedding has captured transferable structure rather than only separations among the training identities.
Limitations
A single distance function presupposes that task relevance can be represented by one shared geometry. This assumption becomes restrictive when similarity depends on context, because the same pair of observations may be close under one relation and distant under another. Conditional metric models address this structure by making the transformation depend on an auxiliary variable, although the resulting comparison function may no longer define a single metric space.
Supervisory relations can also be internally inconsistent. Pair labels may violate transitivity, and separately supplied triplets may contain preference cycles. Margin losses remain numerically definable under such conditions, but no embedding can satisfy all constraints simultaneously. The optimized geometry then represents a compromise determined by loss scaling, regularization, and the frequency with which each relation enters the objective.
The learned representation also inherits information present in the labels and sampling process. If relational supervision reflects systematic measurement differences or unbalanced group coverage, neighborhood structure can reproduce those patterns even when aggregate retrieval performance remains unchanged. This property links metric evaluation to broader questions in algorithmic bias and fairness in machine learning.