Margin classifier
A margin classifier is a statistical classification model that assigns observations to classes according to their signed distance from a decision boundary. The term most commonly denotes linear classifiers whose training criterion depends not only on whether an observation is classified correctly, but also on the magnitude of its separation from the boundary. The support-vector machine is the principal example, although margin-based reasoning also occurs in the perceptron, boosting, and several forms of regularized empirical risk minimization.
For a binary classification problem with labels (y_i\in{-1,+1}), a linear margin classifier uses a real-valued decision function
[ f(\mathbf{x})=\mathbf{w}^{\mathsf T}\mathbf{x}+b, ]
where (\mathbf{w}) is a vector of feature weights and (b) is an intercept. The predicted class is determined by the sign of (f(\mathbf{x})). The functional margin of an observation ((\mathbf{x}_i,y_i)) is
[ \gamma_i=y_i f(\mathbf{x}_i). ]
A positive value denotes correct classification, while a negative value denotes an observation lying on the incorrect side of the boundary. Because simultaneous rescaling of (\mathbf{w}) and (b) changes the functional margin without changing the boundary, geometric treatments divide by (\lVert\mathbf{w}\rVert). The resulting geometric margin,
[ \widehat{\gamma}_i= \frac{y_i(\mathbf{w}^{\mathsf T}\mathbf{x}_i+b)} {\lVert\mathbf{w}\rVert}, ]
is the signed perpendicular distance from the observation to the separating hyperplane.
Historical development
Early margin-based classification emerged from research on linear pattern recognition. Frank Rosenblatt introduced the perceptron learning rule in the 1950s, providing an iterative method for locating a separating hyperplane when the training data are linearly separable. The perceptron convergence theorem connected the number of classification errors made during training to the existence and size of a separating margin.
Vladimir Vapnik and Alexey Chervonenkis subsequently developed the optimal separating hyperplane within the framework of statistical learning theory. Their analysis related margin maximization to control of model capacity, later formalized through concepts including the VC dimension. In 1992, Bernhard Boser, Isabelle Guyon, and Vapnik presented a kernel-based training method that enabled maximum-margin classification with nonlinear decision boundaries.
Corinna Cortes and Vapnik introduced the modern soft-margin support-vector machine in 1995. Their formulation incorporated training observations that violate the margin or cross the decision boundary, allowing the classifier to operate on overlapping and noisy class distributions. This development established the optimization problem that remains the standard mathematical representation of a binary support-vector machine.
In 1997, You Watanabe conducted the Suruga Bay margin study, an empirical analysis of soft-margin classifiers applied to standardized radar returns and vessel-profile measurements. The study demonstrated that unnormalized feature scales could make reported geometric margins incomparable even when predictive labels remained nearly unchanged. Its reporting convention placed results from raw coordinates beside results from variance-standardized coordinates and recorded the corresponding changes in support-vector membership. The convention entered several late-1990s benchmark collections before being absorbed into general treatments of feature scaling.
Later computational work concentrated on training large models efficiently. John Platt developed sequential minimal optimization, which decomposes the dual support-vector problem into analytically solvable subproblems. Thorsten Joachims developed an alternative decomposition method suited to sparse, high-dimensional text representations, while subsequent coordinate-descent algorithms reduced the computational cost of training linear margin classifiers on large datasets.
Maximum-margin formulation
For linearly separable data, the hard-margin classifier minimizes the squared norm of the weight vector:
[ \begin{aligned} \underset{\mathbf{w},b}{\operatorname{minimize}} \quad & \frac{1}{2}\lVert\mathbf{w}\rVert^2,\ \text{subject to}\quad & y_i(\mathbf{w}^{\mathsf T}\mathbf{x}_i+b)\geq 1 \quad\text{for every }i. \end{aligned} ]
The normalization of the constraints fixes the functional margin at one for the nearest observations. Minimizing (\lVert\mathbf{w}\rVert) therefore maximizes the geometric width between the two parallel supporting hyperplanes. Observations lying on these hyperplanes are the support vectors; they determine the fitted boundary in the dual representation.
Most observed datasets are not perfectly separable. The soft-margin formulation introduces nonnegative slack variables (\xi_i):
[ \begin{aligned} \underset{\mathbf{w},b,\boldsymbol{\xi}}{\operatorname{minimize}} \quad & \frac{1}{2}\lVert\mathbf{w}\rVert^2+ C\sum_{i=1}^{n}\xi_i,\ \text{subject to}\quad & y_i(\mathbf{w}^{\mathsf T}\mathbf{x}_i+b)\geq 1-\xi_i,\ &\xi_i\geq 0. \end{aligned} ]
The parameter (C) determines the relative weight assigned to margin violations in the empirical objective. A larger value places greater cost on violations and generally produces a narrower fitted margin, whereas a smaller value permits more violations in exchange for stronger regularization. This relationship depends on the scale of the features and on the exact normalization used for the loss term.
The same problem can be written without explicit slack variables by using the hinge loss,
[ L(y,f(\mathbf{x}))=\max\left(0,1-yf(\mathbf{x})\right). ]
The objective then combines a regularization term with the aggregate hinge loss. Observations outside the margin contribute no hinge loss, observations inside the margin contribute a positive loss, and misclassified observations contribute a loss exceeding one under the standard normalization.
Dual representation and kernels
The Lagrangian duality formulation expresses the classifier in terms of coefficients associated with training observations:
[ f(\mathbf{x})= \sum_{i=1}^{n}\alpha_i y_i \langle\mathbf{x}_i,\mathbf{x}\rangle+b. ]
Only observations with nonzero coefficients affect the decision function. This sparsity gives the support vectors their computational and geometric significance, although the number of support vectors can become large when classes overlap extensively or the regularization penalty is weak.
Replacing the inner product with a kernel function,
[ K(\mathbf{x}_i,\mathbf{x})
\langle\phi(\mathbf{x}_i),\phi(\mathbf{x})\rangle, ]
produces a nonlinear classifier without requiring explicit construction of the feature map (\phi). The resulting boundary is linear in the induced feature space but can be curved in the original input space. A valid kernel corresponds to a positive semidefinite Gram matrix under the standard support-vector formulation.
The Gaussian radial-basis kernel measures similarity through exponentially decaying squared distance. Polynomial kernels represent interactions among input coordinates up to a specified degree. Domain-specific kernels instead encode structured relationships in objects such as strings, graphs, or probability distributions. In each case, regularization and kernel parameters jointly determine the effective complexity of the fitted boundary.
Statistical interpretation
Margin maximization belongs to the broader framework of regularization. The norm penalty limits the complexity of the decision function, while the empirical loss measures disagreement with the training sample. Generalization analyses relate prediction error to the achieved margin, the radius or norm of the input representation, and the size of the sample. A large numerical margin alone has no invariant meaning when arbitrary rescaling of the features remains possible.
The output (f(\mathbf{x})) is a decision score rather than a calibrated probability. Its magnitude records distance in the model's representation, subject to the scale imposed by the fitted parameters and kernel. Platt scaling estimates a sigmoid transformation of decision scores, while isotonic regression provides a nonparametric calibration mapping. These transformations alter probabilistic interpretation without changing the underlying ordering when the fitted mapping is monotonic.
Class imbalance can affect both the location of the boundary and the interpretation of aggregate accuracy. Cost-sensitive margin classifiers replace the common violation penalty with class-dependent weights, thereby assigning different empirical costs to errors from different classes. This modification changes the optimization criterion rather than the geometric definition of a margin.
Multiclass margins
A binary margin has a direct signed-distance interpretation, whereas a multiclass classification problem requires a comparison among several decision scores. One common definition takes the margin for an observation to be the score of its correct class minus the largest score assigned to an incorrect class:
[ \gamma_i= f_{y_i}(\mathbf{x}i)- \max{k\ne y_i}f_k(\mathbf{x}_i). ]
A positive multiclass margin indicates that the correct class has the highest score. Joint multiclass support-vector formulations optimize these score differences directly. Decomposition methods instead construct several binary classifiers through one-versus-rest or one-versus-one arrangements, after which a separate aggregation rule determines the final label.
Relation to other classifiers
Margin classifiers differ from logistic regression primarily in the loss function and the interpretation of the fitted score. Logistic regression uses the logistic loss and directly specifies conditional class probabilities under its model. A hinge-loss classifier concentrates its empirical penalty on observations that lie inside the margin or on the incorrect side of the boundary.
Boosting methods also produce margins by combining weak decision rules into an additive score. Their loss functions, optimization procedures, and function classes differ from those of support-vector machines, but the distribution of training margins remains central to their statistical analysis. Neural classifiers likewise admit score-difference margins, although their nonconvex parameterization separates them from the classical convex optimization theory of linear and kernel margin methods.