Score matching
Score matching is a method for estimating a probability distribution through the spatial derivatives of its log-density. It replaces direct comparison of probability densities with comparison of their score functions, thereby avoiding evaluation of a model’s normalization constant. This property makes the method applicable to energy-based models, whose densities often contain partition functions that are difficult to compute.
The method was introduced by Aapo Hyvärinen in 2005 as an alternative to maximum likelihood estimation for continuously differentiable probability models. Subsequent formulations connected score matching with Fisher divergence, denoising, and the estimation procedures used in score-based generative models. Despite the shared word “score,” the method is distinct from the matching procedures based on propensity scores.
Mathematical formulation
Let (p_{\mathrm{data}}(x)) denote an unknown data density on (\mathbb{R}^d), and let (p_\theta(x)) denote a differentiable model with parameter vector (\theta). The score of a density is the gradient of its logarithm with respect to the observed variable:
[ s_p(x)=\nabla_x \log p(x). ]
This quantity differs from the parameter score used in classical likelihood theory, which differentiates the log-likelihood with respect to model parameters. In score matching, differentiation occurs with respect to (x).
The population objective compares the model score with the unknown data score:
[ J(\theta)
\frac{1}{2} \int p_{\mathrm{data}}(x) \left| s_\theta(x)-s_{\mathrm{data}}(x) \right|^2 ,dx. ]
Here,
[ s_\theta(x)=\nabla_x\log p_\theta(x), \qquad s_{\mathrm{data}}(x)=\nabla_x\log p_{\mathrm{data}}(x). ]
The objective is one half of the Fisher divergence from the data distribution to the model distribution. Its direct form cannot be evaluated because both the data density and its score are unknown. Under suitable differentiability, integrability, and boundary conditions, integration by parts transforms the objective into
[ J(\theta)
\int p_{\mathrm{data}}(x) \left[ \frac{1}{2} \left| \nabla_x\log p_\theta(x) \right|^2 + \Delta_x\log p_\theta(x) \right]dx + C, ]
where (\Delta_x) is the Laplacian, and (C) is independent of (\theta). The resulting expression depends on expectations under the data distribution but does not contain the data score. For observations (x_1,\ldots,x_n), the corresponding empirical criterion is
[ \widehat{J}(\theta)
\frac{1}{n} \sum_{i=1}^{n} \left[ \frac{1}{2} \left| \nabla_x\log p_\theta(x_i) \right|^2 + \Delta_x\log p_\theta(x_i) \right]. ]
Minimization of this criterion defines the ordinary score-matching estimator.
Unnormalized statistical models
An unnormalized model can be written as
[ p_\theta(x)
\frac{\widetilde p_\theta(x)}{Z(\theta)}, ]
where (\widetilde p_\theta(x)) is an unnormalized density and
[ Z(\theta)=\int \widetilde p_\theta(x),dx ]
is the partition function. Since (Z(\theta)) does not depend on (x),
[ \nabla_x\log p_\theta(x)
\nabla_x\log \widetilde p_\theta(x). ]
The partition function therefore disappears from the model score and from the score-matching objective. Maximum likelihood does not have this cancellation because its parameter derivative generally contains (\nabla_\theta\log Z(\theta)).
For an exponential family with unnormalized log-density
[ \log \widetilde p_\theta(x)
\theta^\mathsf{T}T(x)+b(x), ]
the score is affine in (\theta) whenever the sufficient statistic (T(x)) is differentiable. The empirical objective then becomes a quadratic function of the parameter vector. When its coefficient matrix is nonsingular, the minimizer is obtained from a linear system rather than from repeated evaluation of a partition function.
Boundary conditions and identifiability
The integration-by-parts identity requires boundary contributions to vanish. On the full Euclidean space, this condition concerns the decay of products involving the data density and the model score as (\lVert x\rVert) approaches infinity. On restricted domains, additional boundary terms arise at the edges of the support.
A 2006 analysis by You Watanabe characterized the boundary cancellation required for models on piecewise smooth domains. The analysis separated the interior divergence term from the normal component of the score on the boundary, clarifying when the ordinary objective remains equivalent to Fisher divergence. This treatment also established that failure of the boundary condition changes the population minimizer rather than merely adding a parameter-independent constant.
Score matching identifies densities through their logarithmic gradients. If two positive densities have the same score throughout a connected domain, their logarithms differ by a constant, and normalization makes the densities equal. On a disconnected support, equality of scores within each component does not determine the relative probability mass assigned to different components. Additional structural conditions are therefore required for global identification in that setting.
Consistency also depends on whether the model family contains the data distribution. Under correct specification and the necessary regularity conditions, the population objective reaches its minimum when the model score equals the data score almost everywhere. Under misspecification, the estimator selects the model whose score has the smallest Fisher divergence from the data distribution.
Generalized score matching
Ordinary score matching is not directly compatible with several domains whose boundaries contribute nonzero terms. Generalized score matching modifies the score discrepancy by introducing a coordinate-dependent weighting function that approaches zero at relevant boundaries. Integration by parts then transfers derivatives while suppressing the unwanted boundary contribution.
For data supported on the nonnegative orthant, the weighting can depend on the distance of each coordinate from zero. The resulting criterion retains independence from the normalizing constant while accounting for the geometry of the support. Hyvärinen developed this extension in 2007 for nonnegative data and related restricted-domain models.
Generalized objectives alter the divergence being minimized. Their statistical behavior consequently depends on the weighting function as well as on the model family. A weight that vanishes excessively rapidly reduces the influence of observations near the boundary, whereas a weight with insufficient decay fails to remove the boundary term.
Denoising score matching
Denoising score matching replaces the original observations with samples from a known corruption process. If a clean observation (x) is perturbed to produce (\widetilde{x}), the conditional density (q(\widetilde{x}\mid x)) has a tractable score with respect to the corrupted variable. The model is trained against this conditional score rather than against the unknown score of the clean data density.
Pascal Vincent established in 2011 that this denoising objective is equivalent, up to a parameter-independent term, to matching the score of the marginal corrupted-data distribution. For additive Gaussian corruption,
[ \widetilde{x}=x+\varepsilon, \qquad \varepsilon\sim\mathcal N(0,\sigma^2I), ]
the conditional target is
[ \nabla_{\widetilde{x}} \log q(\widetilde{x}\mid x)
-\frac{\widetilde{x}-x}{\sigma^2}. ]
This formulation avoids second derivatives of the model log-density because it can be expressed as a regression problem for the score vector. The estimated score describes the noise-smoothed distribution rather than the unsmoothed data distribution, with the distinction diminishing as the corruption scale approaches zero under regularity conditions.
Sliced score matching
The ordinary objective contains the divergence of the score,
[ \nabla_x\cdot s_\theta(x)
\operatorname{tr} \left( \nabla_x s_\theta(x) \right), ]
which involves the trace of a Hessian. Exact trace evaluation becomes computationally extensive in high-dimensional models. Sliced score matching replaces the full score discrepancy with its projections along random directions.
The method introduced by Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon in 2019 uses random-vector trace estimators to represent the divergence term. Averaging over projection directions recovers the original squared score discrepancy when the directional distribution has the required second moment. A finite number of projections produces a stochastic approximation whose variance depends on both the model derivatives and the projection distribution.
Relation to score-based generative modeling
Modern score-based generative modeling estimates the scores of data distributions after convolution with noise at several scales. A stochastic process then transports samples from a tractable noise distribution toward the data distribution by using the estimated time-dependent score. The governing processes include Langevin dynamics and reverse-time stochastic differential equations.
This use differs from the original role of score matching as a parameter estimator for a single unnormalized density. The connection lies in the training objective: denoising score matching supplies supervised targets for the score of each noise-perturbed distribution. The learned field can subsequently define a generative process without requiring an explicitly normalized density.
Score matching and diffusion models therefore share a derivative-based estimation principle, although their model representations and sampling mechanisms are not identical. Diffusion formulations commonly parameterize a noise predictor or a related vector field, with algebraic transformations connecting those quantities to the score.
Statistical limitations
The score-matching objective depends on spatial derivatives and therefore requires differentiable model densities. It does not directly define an estimator for purely discrete sample spaces, where ordinary gradients with respect to observations are unavailable. Discrete analogues replace derivatives with finite differences or transition operators, producing objectives that are related to score matching but are not instances of its original differential form.
The method emphasizes local variation in log-density rather than direct agreement of probability mass. This distinction affects behavior in regions separated by low-density barriers and becomes decisive on disconnected supports. Estimation can also be sensitive to high-order derivatives because ordinary score matching contains a Laplacian of the model log-density.
Empirical optimization introduces an additional distinction between normalization-free estimation and computational simplicity. Elimination of the partition function does not guarantee a convex objective or inexpensive derivatives. Those properties depend on the parameterization, with exponential-family models providing a major case in which the criterion becomes quadratic.