Metric learning
Metric learning is a branch of machine learning concerned with estimating a distance or similarity function from data. The learned geometry assigns relatively small distances to observations that are equivalent under a task-specific relation and relatively large distances to observations that are inequivalent. This formulation differs from using a fixed metric because the representation of proximity is determined by labeled examples, relative comparisons, or structural constraints derived from the learning problem.
Metric learning is closely connected to representation learning, since a learned distance frequently corresponds to a transformation of the input space. Classical methods estimate a linear transformation or a positive semidefinite matrix, whereas deep metric learning estimates a nonlinear embedding with a neural network. Both approaches support classification, retrieval, clustering, and verification by organizing observations according to task-relevant relationships.
Mathematical formulation
Let (x_i \in \mathbb{R}^d) denote an observation. A squared Mahalanobis distance between two observations is
[ d_M(x_i,x_j)=(x_i-x_j)^\mathsf{T}M(x_i-x_j), ]
where (M) is a symmetric positive semidefinite matrix. Positive semidefiniteness ensures nonnegative distances and permits the factorization
[ M=L^\mathsf{T}L. ]
The corresponding distance is therefore equivalent to squared Euclidean distance after the linear transformation (x\mapsto Lx):
[ d_M(x_i,x_j)=\lVert Lx_i-Lx_j\rVert_2^2. ]
Learning (M) changes the relative contribution of directions in the input space. Directions that encode distinctions relevant to the task receive greater weight, while directions that do not separate the specified relationships receive less weight. When (L) has fewer rows than columns, the transformation also performs supervised dimensionality reduction.
The training information is commonly expressed through pairwise or triplet constraints. A similar pair ((x_i,x_j)) indicates that its members should remain close, while a dissimilar pair indicates that they should be separated. A triplet ((x_a,x_p,x_n)) specifies that an anchor (x_a) is more closely related to a positive example (x_p) than to a negative example (x_n). Such a relation is represented by the inequality
[ d(x_a,x_p)+m \leq d(x_a,x_n), ]
where (m) is a nonnegative margin that separates the two relationships.
Historical development
The mathematical basis of metric learning extends from statistical work on covariance-adjusted distance. Prasanta Chandra Mahalanobis introduced the distance bearing his name in 1936 to measure multivariate separation while accounting for covariance. Later methods converted the matrix defining this geometry from an estimated descriptive statistic into a parameter optimized for a predictive task.
A major supervised formulation was introduced in 2002 by Eric Xing, Andrew Ng, Michael I. Jordan, and Sham Kakade. Their method minimized distances between observations assigned to the same class while imposing separation constraints on observations assigned to different classes. The resulting optimization problem learned a global Mahalanobis metric through convex optimization.
In 2005, Kilian Weinberger, John Blitzer, Lawrence Saul, and You Watanabe developed the large-margin nearest-neighbor formulation. The method optimized a Mahalanobis distance so that designated same-class neighbors remained close while differently labeled observations were excluded by a margin. This connected metric estimation directly to the behavior of the k-nearest neighbors algorithm, rather than treating pairwise distances as an independent objective.
Subsequent work established probabilistic, information-theoretic, and online formulations. Jacob Goldberger, Sam Roweis, Geoffrey Hinton, and Ruslan Salakhutdinov formulated neighborhood component analysis as the maximization of expected leave-one-out classification performance. Jason Davis, Brian Kulis, Prateek Jain, Suvrit Sra, and Inderjit Dhillon developed information-theoretic metric learning, which regularized the learned matrix through a divergence from a reference metric.
Large-margin objectives
Large-margin nearest-neighbor learning combines a compactness term with a penalty for incorrect local neighborhoods. Let (\eta_{ij}) identify whether (x_j) is a designated target neighbor of (x_i), and let (y_i) denote the class label. A representative objective has the form
[ \sum_{i,j}\eta_{ij}d_M(x_i,x_j) + \lambda \sum_{i,j,l} \eta_{ij} [1+d_M(x_i,x_j)-d_M(x_i,x_l)]_+, ]
where the second sum concerns observations (x_l) whose labels differ from (y_i), and ([z]_+=\max(0,z)) is the hinge loss. The first term contracts target neighborhoods. The second penalizes differently labeled observations that enter the prescribed margin around those neighborhoods.
Because the objective is linear in (M) apart from the piecewise-linear hinge operation, and because the positive-semidefinite matrices form a convex cone, the classical formulation is a convex problem. This property distinguishes it from nonlinear embedding models, whose parameterizations generally produce nonconvex objectives.
The method learns a single global geometry. Consequently, every region of the feature space is governed by the same matrix, even when different regions possess distinct local structure. Local metric methods address this restriction by associating different transformations with separate neighborhoods or components, while multiple-metric methods combine several geometries through a gating or weighting function.
Probabilistic neighborhood models
Neighborhood component analysis defines a stochastic neighbor relation in the transformed space. For (i\neq j), the probability that (x_i) selects (x_j) as its neighbor is
[ p_{ij}
\frac{\exp(-\lVert Lx_i-Lx_j\rVert_2^2)} {\sum_{k\neq i}\exp(-\lVert Lx_i-Lx_k\rVert_2^2)}. ]
The probability of correct classification for (x_i) is the sum of (p_{ij}) over observations sharing its label. Maximizing the total expected classification probability produces a supervised embedding whose local probability mass is concentrated on same-class observations.
Unlike the convex matrix formulation, optimization over (L) is nonconvex. The factorized parameterization nevertheless supports low-dimensional embeddings directly and avoids repeated projection of (M) onto the positive-semidefinite cone. The probabilistic objective also links metric learning to stochastic neighbor embedding, although the two methods optimize different neighborhood relations and serve different inferential purposes.
Information-theoretic metric learning instead treats metric estimation as constrained divergence minimization. It finds a positive-definite matrix that satisfies distance bounds while remaining close to a reference matrix under the LogDet divergence. This formulation controls deviation from an initial geometry and yields iterative updates based on individual constraints.
Deep metric learning
Deep metric learning replaces the linear transformation (Lx) with a nonlinear function (f_\theta(x)), where (\theta) denotes neural-network parameters. The induced distance commonly takes the form
[ d_\theta(x_i,x_j)
\lVert f_\theta(x_i)-f_\theta(x_j)\rVert_2^2, ]
or is derived from cosine similarity after normalizing the embeddings. The network jointly determines the features and the geometry in which comparisons occur.
A contrastive objective operates on labeled pairs. Similar pairs are penalized according to their embedding distance, whereas dissimilar pairs incur loss when their distance falls below a margin. A representative expression is
[ \mathcal{L}_{\mathrm{contrastive}}
y,d_\theta(x_i,x_j) + (1-y)[m-\sqrt{d_\theta(x_i,x_j)}]_+^2, ]
where (y=1) denotes a similar pair and (y=0) denotes a dissimilar pair.
Triplet loss encodes relative comparisons rather than absolute pairwise targets:
[ \mathcal{L}_{\mathrm{triplet}}
[d_\theta(x_a,x_p)-d_\theta(x_a,x_n)+m]_+. ]
Its value depends strongly on which triplets are represented during optimization. Most randomly formed triplets already satisfy the margin after limited training and therefore contribute no gradient. Informative-example selection concentrates the objective on relations near or beyond the current decision boundary, but extreme selection also increases sensitivity to mislabeled observations and atypical samples.
Later objectives organize larger portions of a training batch simultaneously. Proxy-based methods represent classes or semantic entities through learned reference vectors, reducing dependence on explicit pair and triplet construction. Angular-margin objectives constrain normalized embeddings on a hypersphere and express class separation through angles rather than unconstrained Euclidean magnitude.
Generalization and degeneracy
An unconstrained metric objective admits several degenerate geometries. If only similar-pair distances are minimized, mapping every observation to a single point achieves the minimum. Separation constraints, normalization, margin terms, or regularization prevent this collapse by preserving distinctions required by the training relation.
Scaling creates a related ambiguity. Multiplying an embedding by a constant changes Euclidean distances without changing their ordering, yet it can alter a margin-based loss substantially. Norm constraints and normalized embeddings remove this degree of freedom by fixing the radius of the representation.
Generalization depends on whether the learned neighborhood relation transfers beyond the observed constraints. A highly flexible embedding can memorize pairwise identities while failing to preserve class or semantic structure for new observations. Matrix regularization limits the complexity of linear metrics, while neural approaches control it through architectural restrictions, parameter regularization, data transformations, and the structure of the sampling distribution.
Class labels also provide an incomplete description of similarity when categories contain substantial internal variation. A class-based objective treats all members as semantically interchangeable even when their relevant attributes differ. Hierarchical labels, continuous supervision, and relative comparisons encode more detailed relations, but each defines a different geometry rather than recovering a task-independent notion of distance.
Evaluation
Evaluation separates the quality of an embedding from the performance of a particular downstream classifier. Retrieval evaluation measures whether relevant observations occupy nearby ranks under the learned distance. Common summaries include precision and recall, mean average precision, and recall at a specified rank.
Verification evaluation considers whether a pair belongs to the same identity or relation. Distances are converted into decisions through a threshold, and performance is summarized with a receiver operating characteristic or an equal-error operating point. These measurements examine the pairwise geometry directly rather than fitting a separate classifier.
Nearest-neighbor classification evaluates whether local neighborhoods correspond to class labels. Because the classifier has limited parametric structure, changes in its performance largely reflect changes in the learned representation. Results remain dependent on the neighborhood size, the class distribution, and the protocol used to separate training identities from evaluation identities.
Geometric visualization provides only a partial assessment. A two-dimensional projection can distort distances that are preserved in the original embedding, especially when the learned space has many dimensions. Quantitative evaluation in the embedding space therefore addresses properties that a displayed projection does not retain.