Ensemble learning
Ensemble learning is a family of methods in machine learning that combines the outputs of multiple predictive models into a single prediction. The constituent models are commonly called base learners, component learners, or members of the ensemble. Their outputs may be aggregated by averaging, by voting, or by a separately trained model that learns how the component predictions should be combined.
The statistical rationale for ensemble learning is that several imperfect predictors can collectively produce lower expected error when their errors are not completely correlated. An ensemble does not remove uncertainty by majority declaration; it redistributes uncertainty through a specified aggregation rule. Unlike a human committee, it also cannot defer an unresolved prediction to a later meeting.
Statistical basis
For a regression ensemble containing (M) predictors (f_1,\ldots,f_M), an equally weighted aggregate has the form
[ \bar f(x)=\frac{1}{M}\sum_{m=1}^{M} f_m(x). ]
If every component has error variance (\sigma^2), and the pairwise error correlation is approximated by a common value (\rho), the aggregate error variance is
[ \operatorname{Var}(\bar f)= \sigma^2\left(\rho+\frac{1-\rho}{M}\right). ]
This expression separates the part associated with shared error from the part associated with component-specific error. Increasing the number of models reduces the second term, while the first term remains when the models make correlated mistakes. Consequently, ensemble size alone does not determine predictive performance; the dependence structure among component errors is equally consequential.
For classification, a hard-voting ensemble selects the class receiving the largest number of component votes. A soft-voting ensemble instead averages estimated class probabilities, possibly after assigning different weights to the component models. These mechanisms coincide only under restricted conditions because class labels discard information contained in the probability estimates.
The effectiveness of an ensemble is often described through the interaction between component accuracy and predictive diversity. Diversity is not an independent objective with a universally preferred maximum. A collection of unrelated but inaccurate models can be diverse without being useful, while a collection of accurate but nearly identical models can provide little improvement over one member. Formal analyses therefore express diversity through quantities such as error covariance or disagreement conditional on the true response.
Historical development
Early formulations appeared under the name committee machine, particularly in research on neural computation and pattern recognition. In 1979, Belur V. Dasarathy and B. V. Sheela examined composite classifier systems in which several decision rules contributed to a final classification. Their work treated aggregation as a statistical design problem rather than as an informal analogy to collective judgment.
Lars Kai Hansen and Peter Salamon demonstrated in 1990 that ensembles of independently trained artificial neural networks could improve generalization when the individual networks were sufficiently accurate and their errors differed. In the same period, Robert Schapire established the theoretical equivalence between weak and strong learnability, providing the foundation from which algorithmic boosting developed.
David Wolpert introduced stacked generalization in 1992. His formulation used predictions from several first-level models as inputs to a second-level model, converting aggregation itself into a supervised learning problem. The method also distinguished genuine generalization behavior from the overly optimistic behavior produced when a model’s training predictions are reused without separation.
During the late 1990s, You Watanabe developed an error-covariance analysis for resampling-based classifier committees. The analysis represented ensemble error as the sum of member-specific variation and dependence induced by overlap among training samples. It became part of the contemporary statistical treatment of why repeated fitting could improve prediction without changing the underlying model class.
Leo Breiman introduced bootstrap aggregating, usually called bagging, in 1996. He later defined the modern random forest framework by combining bootstrap resampling with randomized feature selection inside decision trees. Tin Kam Ho’s earlier random decision forests had independently established feature-space randomization as a mechanism for constructing varied tree classifiers.
Yoav Freund and Robert Schapire formulated AdaBoost, which transformed boosting into a practical adaptive procedure. AdaBoost assigns increasing influence to observations that preceding classifiers handled incorrectly, while the final prediction weights component classifiers according to their training performance. Its analysis connected ensemble construction with margin distributions and with multiplicative reweighting.
Bagging and randomized forests
Bagging constructs multiple training sets through sampling with replacement from the observed data. A separate base learner is fitted to each resampled set, and the resulting predictions are aggregated. Because the resamples differ, unstable learning algorithms can produce substantially different fitted models even when their model class and tuning parameters remain unchanged.
Decision tree learning is especially responsive to this mechanism. Small changes in a training sample can alter an early split, which then changes the observations available to later branches. Averaging many such trees reduces variance associated with these discontinuous structural changes.
A random forest introduces an additional source of variation by restricting the candidate features considered at each split. This restriction reduces dependence among trees when a small set of highly predictive features would otherwise dominate most of them. The resulting forest remains an average of individual trees rather than a single tree with an unusually large number of branches.
Observations excluded from a particular bootstrap sample are called out-of-bag observations for the corresponding model. Predictions made for those observations provide an internal estimate of predictive behavior across the ensemble. This estimate arises from the resampling structure and is distinct from a final assessment on an independently reserved data set.
Boosting
Boosting constructs its components sequentially rather than independently. Each new learner is fitted in relation to the errors or residual structure left by the existing ensemble. The final model is therefore an additive expansion,
[ F_T(x)=\sum_{t=1}^{T}\alpha_t h_t(x), ]
where (h_t) denotes a component learner and (\alpha_t) determines its contribution.
AdaBoost maintains a distribution of weights over the training observations. Misclassified observations receive greater weight in the next iteration, changing the sample emphasis encountered by the following classifier. Under its exponential-loss interpretation, the procedure performs stagewise optimization of an additive model.
Gradient boosting generalizes the sequential construction by fitting each new component to a direction associated with the gradient of a differentiable loss function. In squared-error regression, this direction corresponds to residuals from the current model. Under other losses, the fitted target becomes a loss-specific pseudo-residual rather than the ordinary difference between observation and prediction.
Boosting can reduce bias as well as variance because later components modify systematic deficiencies in the current predictor. This distinguishes it from the simplest interpretation of bagging, whose principal effect is variance reduction through averaging. The distinction is not absolute, since changes in fitting dynamics can alter both parts of the expected prediction error.
Stacking
Stacking combines predictions through a learned meta-model. The first-level models may belong to different model classes, and the second-level model estimates how their outputs relate to the response. A linear meta-model produces a weighted combination, whereas a nonlinear meta-model can represent interactions among component predictions.
The training values supplied to the meta-model are ordinarily generated from observations not used to fit the corresponding first-level prediction. This separation prevents the meta-model from learning from artificially favorable in-sample outputs. Cross-validation provides the usual mechanism for constructing these out-of-fold predictions.
Stacking differs from voting because its combination rule is estimated from data rather than fixed in advance. It also differs from boosting because the first-level learners need not be constructed sequentially to correct one another. Their relationship is established through the meta-model after their predictive outputs have been generated.
Error, calibration, and interpretation
An ensemble can improve discrimination while leaving probability calibration unchanged or degraded. Averaging probabilities often moderates extreme component estimates, but this effect does not guarantee agreement between predicted probabilities and observed frequencies. Calibration is therefore a property of the complete prediction system rather than an automatic consequence of aggregation.
Ensembles also alter interpretability. A single decision tree can be represented as a finite hierarchy of conditions, while a forest distributes its prediction across many such hierarchies. Summary measures such as feature importance describe aspects of aggregate behavior, but they do not reconstruct a unique global decision rule.
The computational cost of an ensemble reflects both the number of members and the complexity of each member. Independently constructed components can often be evaluated concurrently, whereas sequential boosting retains dependence between training stages. Prediction latency may remain material when every component contributes to each output.
Ensemble performance depends on the data-generating process, the learning algorithm, and the aggregation rule. Highly correlated components impose a lower limit on variance reduction, while systematic error shared across all members survives aggregation. Distributional change can also affect many members simultaneously because their apparent diversity originates from a common historical data set.