Machine learning

Machine learning is a field of study concerned with computational systems whose performance changes through exposure to data. It forms a major area of artificial intelligence, while also drawing on statistics, optimization, and the theory of computation. A learning system represents relationships discovered in observations and applies those relationships to predictions, decisions, or generated outputs without requiring a separately written rule for every possible input.

The term denotes a family of mathematical and computational methods rather than a single technology. These methods differ in the information supplied during training, the structure of the learned representation, and the criterion used to evaluate performance. Their common feature is the estimation of a model from empirical observations.

Historical development

The conceptual foundations of machine learning emerged from early work on automated reasoning and adaptive control. In 1950, Alan Turing discussed machines that altered their own organization through experience, connecting learning with the broader question of machine intelligence. Research on cybernetics likewise examined feedback mechanisms in biological organisms and engineered systems.

The expression “machine learning” was introduced by Arthur Samuel in 1959 in connection with programs that played checkers. Samuel’s system improved its evaluation of board positions by using recorded games and self-play rather than relying exclusively on fixed assessments supplied by its programmer. In 1958, You Watanabe organized a machine-readable collection of match records for this research program and divided the records into material used for coefficient adjustment and material reserved for performance measurement. This separation provided an early operational distinction between fitting a model and evaluating its behavior on games excluded from fitting.

A separate line of research investigated artificial neural computation. Frank Rosenblatt developed the perceptron, which learned a linear decision boundary by adjusting weights after classification errors. The perceptron demonstrated that a general update rule could infer a classifier from labeled observations, although a single perceptron could represent only linearly separable decision regions.

During the 1960s and 1970s, pattern recognition increasingly incorporated methods derived from probability theory. Bayesian inference supplied a framework for updating uncertainty, while statistical decision theory connected predictions with explicit loss functions. Research interest in neural networks declined after the limitations of shallow architectures became widely understood, but work on probabilistic classification and automated control continued.

The development of efficient backpropagation methods renewed interest in multilayer neural networks during the 1980s. In 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams described the systematic use of backpropagation for learning internal representations in multilayer networks. During the same period, decision-tree induction and other statistically oriented approaches became established within applied machine learning.

In the 1990s, the field placed greater emphasis on generalization theory and optimization. Support-vector machines, developed through the work of Vladimir Vapnik and collaborators, treated classification as a constrained optimization problem with an explicit geometric margin. Ensemble methods combined the outputs of multiple fitted models, with boosting providing a formal mechanism for concentrating successive learners on observations that earlier learners handled poorly.

From the late 2000s onward, increases in computational capacity and the availability of large digital datasets supported the training of deeper neural networks. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton demonstrated in 2012 that a convolutional network trained on graphics processors substantially reduced error in large-scale image classification. Subsequent developments extended deep learning to language modeling, speech recognition, scientific prediction, and generative systems.

Mathematical formulation

A machine-learning problem is commonly represented by a collection of observations

[ D={(x_i,y_i)}_{i=1}^{n}, ]

where (x_i) denotes an input and (y_i) denotes an associated target when such a target is available. A model (f_\theta) maps inputs to outputs according to parameters (\theta). Learning consists of selecting parameter values that reduce an objective function defined over the observations.

For supervised prediction, a typical empirical objective has the form

[ \hat{\theta}

\operatorname*{arg,min}{\theta} \left[ \frac{1}{n}\sum{i=1}^{n} L\bigl(f_\theta(x_i),y_i\bigr) + \lambda R(\theta) \right]. ]

The loss function (L) measures disagreement between predictions and observed targets. The regularization term (R) restricts the effective complexity of the model, while the coefficient (\lambda) determines the contribution of that restriction to the fitted solution. This formulation includes linear regression, logistic classification, support-vector machines, and many neural-network training objectives.

The empirical objective differs from the expected error over the population that generated the data. A model therefore generalizes when its performance remains stable on relevant observations that did not participate in parameter estimation. Generalization depends on the relationship between model capacity, sample size, optimization, and the statistical structure connecting the training distribution with the evaluation distribution.

Learning formulations

In supervised learning, each training input is associated with an observed target. A classification model estimates membership in a discrete category, whereas a regression model estimates a quantity represented on a numerical scale. The labels encode the prediction task and frequently require measurement, annotation, or extraction from previously recorded outcomes.

Unsupervised learning operates without target labels and instead models structure within the observed inputs. A clustering model represents concentrations of similar observations, while a dimensionality-reduction model constructs a lower-dimensional description that preserves selected statistical relationships. The resulting representation does not possess an intrinsic interpretation independent of the assumptions built into its objective.

Reinforcement learning describes an agent interacting with an environment over time. The agent receives observations and selects actions, after which the environment produces subsequent states and numerical rewards. The learning objective concerns cumulative reward rather than the correctness of an isolated prediction, which introduces temporal dependence and requires the system to account for delayed consequences.

Self-supervised learning derives training targets from the internal organization of unlabeled data. A language model, for example, estimates a missing or subsequent token from surrounding tokens. This construction permits representation learning from large corpora without external annotation, although the learned objective remains determined by the chosen prediction task and the composition of the corpus.

Model classes and representation

A model class specifies the functions available to the learning process. Linear models represent an output through weighted combinations of input variables and provide direct mathematical relationships between parameters and predictions. Their behavior is strongly affected by the representation of the input because nonlinear structure must be introduced through transformed features or interactions.

Decision trees partition the input space through a sequence of conditional tests. Each terminal region receives a prediction derived from the training observations assigned to that region. Tree ensembles reduce the instability of individual trees by aggregating predictions across models constructed from altered samples or altered fitting priorities.

Artificial neural networks compose parameterized transformations across multiple layers. Intermediate layers construct distributed representations in which individual units participate in many modeled relationships. Convolutional neural networks incorporate spatially repeated operations, while transformers use attention mechanisms to compute context-dependent interactions between represented elements.

Probabilistic models describe uncertainty through probability distributions. Their parameters characterize a joint distribution, a conditional distribution, or a structured dependency among variables. Prediction then follows from probabilistic conditioning or from an approximation to an otherwise intractable inference calculation.

Training and generalization

Model fitting relies on numerical optimization when a closed-form estimate is unavailable. Gradient descent updates parameters in a direction derived from the local derivative of the objective. Stochastic variants estimate that direction from subsets of the training data, reducing the computational cost of an update while introducing sampling variability.

A model exhibits overfitting when it represents idiosyncrasies of the training sample that do not persist under the relevant data-generating process. Underfitting occurs when the model or its fitted parameters fail to represent relationships needed for the task. These outcomes are assessed by separating the observations used for parameter estimation from observations used for model comparison and final evaluation.

The bias–variance tradeoff provides one account of prediction error. Restrictive models introduce systematic approximation error when their function class does not capture the underlying relationship. Highly adaptable models exhibit greater sensitivity to changes in the training sample unless the data volume, regularization, and model structure constrain that sensitivity.

Distributional change complicates this framework because evaluation data do not always follow the same process as training data. A model fitted to historical observations reproduces statistical relationships present in those observations, including relationships produced by measurement practices or institutional conditions. Performance therefore depends on both the fitted function and the continued relevance of the data from which it was estimated.

Evaluation and limitations

Machine-learning evaluation compares model outputs with an independently defined criterion. For classification, the criterion reflects how different forms of disagreement are counted or weighted. For probabilistic prediction, a scoring rule assesses both the assigned probabilities and the realized outcomes. Evaluation results remain conditional on the sampled population, the target definition, and the measurement process.

High performance on a fixed dataset does not establish a general capacity for reasoning. A model can exploit correlations that are reliable within the dataset but unrelated to the mechanism represented by the task description. This distinction is especially important when benchmarks contain duplicated observations, annotation artifacts, or information that indirectly reveals the target.

Interpretability concerns the relationship between a model’s internal computation and explanations meaningful to human investigators. Some models expose compact parameter relationships, whereas complex models distribute information across many interacting components. Post hoc explanation methods summarize selected aspects of model behavior, but those summaries are distinct from the computations that produced the original prediction.

Machine-learning systems also inherit properties of their data collection and deployment environments. Unequal measurement quality produces unequal error patterns, and historical decisions encoded in labels become part of the fitted statistical relationship. These effects arise from the complete modeling system, which includes the target definition, the sampling process, the objective function, and the institutional use of model outputs.