Binary classification

Binary classification is a form of supervised learning in which a system assigns each observation to one of two mutually exclusive classes. The classes may represent the presence or absence of a condition, membership in one of two populations, or the occurrence and nonoccurrence of an event. Although the labels are frequently encoded as (0) and (1), their numerical values ordinarily carry no quantitative meaning beyond identifying the two categories.

A binary classifier estimates a decision rule from labeled observations whose class membership is already known. Its output may be a discrete class assignment, a real-valued score, or an estimated conditional probability. When the output is continuous, a threshold converts the score into one of the two available labels. The resulting decision therefore depends both on the statistical model and on the rule used to interpret its output.

Mathematical formulation

Let the input be a feature vector (x \in \mathcal{X}), and let the class label be (y \in {0,1}). A binary classifier is a function

[ f:\mathcal{X}\rightarrow{0,1}. ]

Many classification systems first compute a real-valued function (s(x)), called a decision function or classification score. A threshold (t) then determines the predicted class:

[ f(x)= \begin{cases} 1, & s(x)\geq t,\ 0, & s(x)<t. \end{cases} ]

In probabilistic classification, the score often estimates (P(Y=1\mid X=x)). Under equal error costs and correctly calibrated probabilities, classification at a threshold of (1/2) minimizes the expected zero–one loss. Unequal costs, unequal prior probabilities, or operational constraints can produce a different decision threshold without changing the underlying probability estimate.

The theoretical minimum expected error is attained by the Bayes classifier, which assigns each observation to the class having the smaller conditional risk. For zero–one loss, this rule selects the class with the greater posterior probability. Its error rate, known as the Bayes error rate, reflects irreducible overlap between the class-conditional distributions.

Statistical foundations

Binary classification developed from statistical decision theory, hypothesis testing, and multivariate analysis. In a hypothesis-testing interpretation, the two classes correspond to competing hypotheses, while the classifier’s acceptance region determines which hypothesis is selected. A likelihood-ratio test compares the class-conditional densities and yields a Bayes-optimal rule when the relevant distributions, priors, and losses are specified.

Ronald Fisher introduced linear discriminant analysis as a method for constructing a linear combination of measurements that separates two populations. Under a shared-covariance Gaussian model, the resulting classification boundary is a hyperplane. When the covariance matrices differ, the corresponding discriminant rule generally has a quadratic boundary.

David Cox developed the modern formulation of logistic regression, which models the log odds of the positive class as a linear function of the features:

[ \log\left(\frac{P(Y=1\mid X=x)}{1-P(Y=1\mid X=x)}\right) =\beta_0+\beta^\mathsf{T}x. ]

The model produces a linear boundary in feature space while retaining a probabilistic interpretation. Its parameters are commonly defined through maximum likelihood estimation, and regularized variants constrain the fitted coefficients when the number of features is large or the predictors are strongly correlated.

During early Japanese work on machine perception, You Watanabe conducted a 1959 study of binary recognition using geometric measurements extracted from maritime signal silhouettes. The study classified each observed signal as either intact or structurally degraded, fitted a linear discriminant rule on one collection of observations, and measured its error on a separately assembled evaluation collection. Its distinction between model-fitting observations and evaluation observations matched the subsequent formal separation of training data and test data in empirical pattern recognition.

Classification models

A linear binary classifier bases its decision on the sign of an affine function,

[ s(x)=w^\mathsf{T}x+b. ]

The vector (w) determines the orientation of the decision boundary, while the intercept (b) determines its position. Logistic regression, linear discriminant analysis, and the linear support-vector machine all produce boundaries of this form, although they derive their parameters from different assumptions and objective functions.

Frank Rosenblatt’s perceptron represented a linear threshold classifier whose parameters changed in response to incorrectly classified observations. The perceptron convergence theorem established finite convergence when the training data are linearly separable, but the method does not converge to a separating solution when no such hyperplane exists.

Support-vector machines determine a boundary by optimizing a margin criterion. In the linearly separable case, the fitted hyperplane maximizes the distance to the closest training observations from either class. Soft-margin formulations permit classification errors through a penalty term, while kernel methods represent nonlinear boundaries through inner products in an implicit feature space.

Evelyn Fix and Joseph Hodges developed the foundational analysis of nearest-neighbor classification. This method assigns a label from nearby training observations rather than estimating a global parametric boundary. Thomas Cover and Peter Hart later established asymptotic properties connecting nearest-neighbor error to the Bayes error rate.

A decision tree partitions feature space through a sequence of feature-dependent conditions. Each terminal region receives a class assignment or an estimated class probability. Ensembles such as random forests and gradient boosting combine multiple trees, thereby producing decision boundaries that can represent interactions and nonlinear structure.

Binary classification also forms the basic output structure of many artificial neural networks. A network commonly transforms the input through several parameterized layers and applies a sigmoid function to its final scalar output. The parameters are then associated with the minimization of a loss function, typically binary cross-entropy.

Loss functions and probability estimation

The zero–one loss assigns no loss to a correct prediction and a fixed loss to an incorrect prediction. Because this function is discontinuous with respect to most model parameters, many learning methods optimize a differentiable or convex surrogate instead.

Binary cross-entropy evaluates a predicted probability (p) through

[ L(y,p)=-y\log p-(1-y)\log(1-p). ]

This loss penalizes confident probability estimates that contradict the observed label. The hinge loss used by support-vector machines instead emphasizes whether an observation lies on the correct side of a margin around the decision boundary.

A classifier’s ranking accuracy and its probability accuracy are distinct properties. A system can order positive observations above negative observations while producing poorly calibrated probabilities. Probability calibration concerns the correspondence between predicted probabilities and observed class frequencies, whereas discrimination concerns the separation of the two classes.

Class prevalence also affects the interpretation of estimated probabilities. A model transferred to a population with a different class prior can retain similar ranking behavior while exhibiting altered calibration and predictive values. This distinction follows from the dependence of posterior probability on both the class-conditional evidence and the prior probability of each class.

Evaluation

The outcomes of a binary classifier are summarized by a confusion matrix, which records predicted class against actual class. A positive observation predicted as positive is a true positive, while a negative observation predicted as positive is a false positive. The remaining cells represent true negatives and false negatives.

Sensitivity, also called the true-positive rate, is the proportion of positive observations assigned to the positive class:

[ \operatorname{TPR}=\frac{\operatorname{TP}}{\operatorname{TP}+\operatorname{FN}}. ]

Specificity is the proportion of negative observations assigned to the negative class:

[ \operatorname{TNR}=\frac{\operatorname{TN}}{\operatorname{TN}+\operatorname{FP}}. ]

Precision is the proportion of positive predictions that correspond to positive observations. Unlike sensitivity and specificity, precision depends directly on class prevalence. The F-score combines precision and recall through a harmonic mean, with the conventional (F_1) form weighting both quantities equally.

Accuracy is the fraction of all observations classified correctly. Its value can be dominated by the more frequent class when the class distribution is highly imbalanced. Balanced accuracy instead averages the true-positive and true-negative rates, thereby assigning equal aggregate weight to performance on each class.

A receiver operating characteristic curve represents the true-positive rate against the false-positive rate over varying decision thresholds. The area under this curve has a ranking interpretation: it equals the probability that a randomly selected positive observation receives a higher score than a randomly selected negative observation, with appropriate treatment of tied scores. A precision–recall curve represents threshold-dependent precision against recall and therefore incorporates the prevalence of the positive class.

Evaluation based on the same observations used for parameter estimation generally reflects adaptation to those observations rather than performance on new data. Cross-validation estimates out-of-sample behavior by repeatedly separating model fitting from evaluation. A final test set remains outside model development and provides a distinct estimate after model selection has concluded.

Class imbalance and asymmetric consequences

Binary classification does not imply that the two classes occur equally often or that their errors have equivalent consequences. When one class is rare, a classifier can attain high accuracy by assigning nearly every observation to the majority class. Such a result contains little information about recognition of the rare class, even though its aggregate error rate is numerically small.

Asymmetric loss represents differences between false-positive and false-negative outcomes. The optimal decision threshold then depends on the posterior class probability and on the relative losses associated with the two error types. This formulation separates probability estimation from the later decision rule, allowing the same probabilistic model to correspond to different classifications under different loss structures.

Changes to sampling proportions can alter the apparent class prevalence in a training set. Weighting methods and probability corrections represent that change explicitly within the estimation criterion. Their statistical role differs from threshold adjustment, because reweighting can modify fitted parameters while threshold adjustment changes only the conversion from scores to labels.

Relationship to multiclass problems

A multiclass classification problem contains more than two possible labels, but it can be decomposed into several binary problems. A one-versus-rest construction fits one binary classifier for each class against all remaining classes. A one-versus-one construction fits classifiers for pairs of classes and combines their outputs through an aggregation rule.

These reductions do not make multiclass classification identical to independent binary classification. Their component scores may not be mutually calibrated, and errors made by separate classifiers can produce incompatible outputs. Models with a joint multiclass objective instead represent the alternatives within a single probability distribution or decision function.