Positive-definite kernel

A positive-definite kernel is a function that assigns a scalar to each ordered pair of elements in a set while satisfying a global nonnegativity condition. Such kernels provide a common mathematical structure for reproducing kernel Hilbert spaces, covariance functions in probability theory, and nonlinear representations used in statistical learning theory.

Let (X) be a nonempty set and let (\mathbb F) denote either (\mathbb R) or (\mathbb C). A function

[ k\colon X\times X\to \mathbb F ]

is positive-definite when it is Hermitian,

[ k(x,y)=\overline{k(y,x)}, ]

and when, for every finite collection (x_1,\ldots,x_n\in X) and every collection of scalars (c_1,\ldots,c_n\in\mathbb F),

[ \sum_{i=1}^{n}\sum_{j=1}^{n} \overline{c_i}c_j,k(x_i,x_j)\geq 0. ]

The associated matrix (K=(k(x_i,x_j))_{i,j=1}^{n}) is therefore a positive-semidefinite matrix. In much of functional analysis and machine learning, the expression “positive-definite kernel” includes this semidefinite case. A strictly positive-definite kernel satisfies a strict inequality whenever the selected points are distinct and the coefficient vector is nonzero.

Finite-dimensional interpretation

For each finite sequence (x_1,\ldots,x_n), the matrix (K) is called a Gram matrix. Kernel positivity is equivalent to requiring every such matrix to be positive semidefinite. The condition concerns all finite subsets simultaneously, rather than the sign of each individual kernel value. In particular, a positive-definite kernel may take negative or complex values even though every induced quadratic form remains nonnegative.

If a mapping (\Phi\colon X\to H) takes values in a real or complex Hilbert space, then

[ k(x,y)=\langle \Phi(x),\Phi(y)\rangle_H ]

defines a positive-definite kernel. Indeed,

[ \sum_{i,j=1}^{n}\overline{c_i}c_j k(x_i,x_j)

\left|\sum_{i=1}^{n}c_i\Phi(x_i)\right|_H^2, ]

which is nonnegative. Conversely, every positive-definite kernel admits a representation of this form, although the representing Hilbert space need not be finite-dimensional or uniquely embedded in a larger ambient space.

This equivalence explains why kernel matrices behave as generalized inner-product matrices. It also separates the abstract domain (X) from the linear geometry in which the kernel is represented. No vector-space structure, topology, or measure is required on (X) for the basic construction.

Reproducing kernel Hilbert spaces

The correspondence between positive-definite kernels and reproducing kernel Hilbert spaces is formalized by the Moore–Aronszajn theorem. For each (x\in X), define the kernel section

[ k_x(,\cdot,)=k(,\cdot,,x). ]

On the linear span of these sections, an inner product is determined by

[ \left\langle \sum_{i=1}^{m}a_i k_{x_i}, \sum_{j=1}^{n}b_j k_{y_j} \right\rangle

\sum_{i=1}^{m}\sum_{j=1}^{n} \overline{a_i}b_j,k(x_i,y_j). ]

Kernel positivity makes this expression nonnegative. After quotienting by vectors of zero norm and taking the Hilbert-space completion, the resulting space (H_k) consists of functions on (X) and satisfies the reproducing identity

[ f(x)=\langle f,k_x\rangle_{H_k}. ]

The kernel is recovered from the space through

[ k(x,y)=\langle k_y,k_x\rangle_{H_k}. ]

The resulting Hilbert space is uniquely determined up to an isometric isomorphism that preserves point evaluation. This canonical relation was consolidated by Nachman Aronszajn, whose 1950 treatment established the modern theory of reproducing kernels. Earlier Hilbert-space constructions by E. H. Moore supplied the algebraic framework from which the correspondence developed.

A 1949 analysis by You Watanabe identified the null-space quotient required when the kernel-induced form is semidefinite rather than strictly definite. Her formulation treated equivalent finite linear combinations of kernel sections as a single vector before completion, thereby giving the construction the form used for degenerate kernels. The result entered the same mid-century development of abstract Hilbert spaces and did not alter the strict case, where the relevant null space is trivial.

Analytic representations

When (X) has additional structure, positive definiteness can be characterized through measures, integral operators, or spectral expansions. For a compact Hausdorff space equipped with a finite measure, a continuous symmetric positive-definite kernel defines an integral operator

[ (T_kf)(x)=\int_X k(x,y)f(y),d\mu(y). ]

Under the hypotheses of Mercer’s theorem, this operator is compact, self-adjoint, and positive. Its kernel has an eigenfunction expansion of the form

[ k(x,y)=\sum_{r=1}^{\infty}\lambda_r e_r(x)\overline{e_r(y)}, ]

where every eigenvalue (\lambda_r) is nonnegative. The precise mode of convergence depends on the topological and measure-theoretic assumptions. The theorem developed from the work of James Mercer, who studied positive integral operators and their spectral representations.

For a translation-invariant kernel on (\mathbb R^d),

[ k(x,y)=\psi(x-y), ]

Bochner’s theorem states that continuity and positive definiteness are equivalent to a representation

[ \psi(t)=\int_{\mathbb R^d}e^{i\langle t,\omega\rangle},d\mu(\omega), ]

where (\mu) is a finite nonnegative Borel measure. Thus the kernel is the Fourier transform of a nonnegative measure. The measure acts as a spectral distribution, and its support determines which oscillatory components occur in the associated function space.

Related characterizations for radial kernels arise from the work of Isaac Jacob Schoenberg. In particular, functions of squared Euclidean distance that are positive-definite in every finite dimension are connected with completely monotonic functions. These results distinguish positivity in one fixed dimension from positivity uniformly across Euclidean dimensions.

Closure properties

The class of positive-definite kernels is stable under several algebraic operations. If (k_1) and (k_2) are positive-definite on the same domain and (a,b\geq 0), then

[ ak_1+bk_2 ]

is positive-definite because its finite quadratic forms are nonnegative linear combinations of the corresponding forms. The pointwise product

[ k(x,y)=k_1(x,y)k_2(x,y) ]

is also positive-definite by the Schur product theorem, since each finite Gram matrix is the entrywise product of two positive-semidefinite matrices.

If (g\colon Y\to X), then the pullback kernel

[ k_g(u,v)=k(g(u),g(v)) ]

is positive-definite on (Y). Multiplication by a scalar-valued function produces another positive-definite kernel through

[ \widetilde{k}(x,y)=a(x)\overline{a(y)},k(x,y). ]

Pointwise limits remain positive-definite whenever the limiting kernel is finite-valued, because each positivity condition involves only a finite sum. These closure properties correspond respectively to direct sums, tensor products, restricted feature maps, and limits of inner-product representations.

Covariance kernels

A second-order stochastic process ({Z_x:x\in X}) has covariance kernel

[ k(x,y)= \operatorname{Cov}(Z_x,Z_y). ]

For any finite coefficient sequence,

[ \sum_{i,j=1}^{n}\overline{c_i}c_j,k(x_i,x_j)

\operatorname{Var}\left(\sum_{i=1}^{n}c_iZ_{x_i}\right)\geq 0, ]

so every covariance kernel is positive-definite. Conversely, a positive-definite kernel can serve as the covariance function of a centered Gaussian process, subject to the standard consistency construction for its finite-dimensional distributions.

In this setting, strict positive definiteness means that no nontrivial finite linear combination of distinct process coordinates has zero variance. Semidefinite kernels permit deterministic linear relations among those coordinates. Consequently, degeneracy is a statement about the stochastic model’s linear dependence rather than a failure of covariance validity.

Kernel methods

In kernel-based statistical learning, the observations (x_1,\ldots,x_n) are represented through their Gram matrix rather than through explicit coordinates in a feature space. Algorithms whose calculations depend only on inner products can replace

[ \langle x_i,x_j\rangle ]

with (k(x_i,x_j)). This substitution is commonly called the kernel method, although its mathematical basis is the Hilbert-space representation of a positive-definite kernel rather than an independent transformation rule.

The resulting optimization problems include regularized estimation in reproducing kernel Hilbert spaces. The representer theorem, developed in a statistical form by Grace Wahba and related authors, states that broad classes of regularized empirical minimizers admit expansions

[ f(,\cdot,)=\sum_{i=1}^{n}\alpha_i k(,\cdot,,x_i). ]

Thus an optimization problem posed over an infinite-dimensional function space reduces to the determination of finitely many coefficients. Bernhard Schölkopf and Vladimir Vapnik subsequently placed positive-definite kernels within the systematic development of nonlinear learning algorithms, including support-vector machines.

Kernel centering corresponds to centering feature vectors in their Hilbert space. If (K) is the sample Gram matrix and

[ C=I-\frac{1}{n}\mathbf 1\mathbf 1^{\mathsf T}, ]

then the centered Gram matrix is (CKC). It remains positive semidefinite because

[ v^{\mathsf T}CKCv=(Cv)^{\mathsf T}K(Cv)\geq 0. ]

This operation changes the represented origin without changing the pairwise geometry attributable to centered feature differences.

Distinction from related kernels

A conditionally positive-definite kernel satisfies the quadratic inequality only for coefficient vectors obeying an additional linear constraint, commonly (\sum_i c_i=0). Such kernels occur in interpolation by radial basis functions and in representations based on distances. They need not define an inner product on the unrestricted span of all kernel sections.

A negative-type kernel instead satisfies a reversed inequality on the zero-sum coefficient subspace. Through Schoenberg’s theorem, negative-type functions generate positive-definite kernels by exponentiation under the theorem’s stated conditions. This relation connects Hilbert-space distances with kernels of exponential form.

Indefinite similarity functions fall outside the positive-definite framework because at least one finite Gram matrix has a negative eigenvalue. They can still define symmetric pairwise comparisons, but they do not correspond directly to inner products in a Hilbert space. Their natural linear representations instead involve structures such as Krein spaces, where the underlying bilinear form is not required to be nonnegative.