Calibration (statistics)
In statistics, calibration is the agreement between probabilistic predictions and the empirical frequencies or conditional distributions of the quantities being predicted. A forecasting system is calibrated when events assigned a given probability occur with that probability in the relevant population. Calibration therefore concerns the statistical interpretation of a prediction rather than the sharpness, discrimination, or overall accuracy of the forecasting system.
For a binary outcome (Y\in{0,1}) and a forecast probability (P), exact calibration is expressed as
[ \Pr(Y=1\mid P=p)=p ]
for every forecast value (p) having positive probability. Equivalently,
[ \mathbb{E}[Y\mid P]=P. ]
This condition states that, among cases receiving a forecast near (0.7), the limiting proportion with outcome (Y=1) is (0.7). It does not require those cases to be otherwise similar, nor does it determine how frequently the forecasting system assigns probabilities near (0), (0.5), or (1).
Calibration is studied in probability forecasting, statistical classification, regression analysis, and the assessment of predictive distributions. Its interpretation depends on the forecast object and on the population with respect to which the conditional relationship is defined.
Binary probability forecasts
For binary events, the calibration function is
[ c(p)=\Pr(Y=1\mid P=p). ]
A perfectly calibrated system has (c(p)=p). The difference (c(p)-p) is sometimes called the conditional calibration error. Because continuously valued forecasts rarely repeat exactly in a finite sample, empirical analyses estimate the calibration function by grouping nearby probabilities or by applying a statistical smoother.
A reliability diagram displays estimated event frequencies against forecast probabilities. The diagonal line represents exact calibration. A curve above the diagonal indicates that the observed event frequency exceeds the stated probability, while a curve below the diagonal indicates the reverse relationship. The sampling variability of the displayed frequencies depends on the number and distribution of observations within each part of the forecast range.
Calibration does not imply that a forecast is informative. A constant forecast equal to the unconditional event frequency is calibrated whenever the evaluation population has the same event frequency. Such a forecast provides no case-specific separation between events and non-events. Conversely, a forecast can discriminate strongly between the two outcome classes while assigning probabilities that are systematically too high or too low.
This distinction is reflected by the relationship between calibration and the receiver operating characteristic. The receiver operating characteristic depends on the ranking induced by forecast scores and is unchanged by strictly increasing transformations. Calibration depends on the numerical values of the probabilities and generally changes under the same transformations.
Multiclass and conditional calibration
For an outcome taking one of (K) categories, a probabilistic forecast is a vector
[ \mathbf{P}=(P_1,\ldots,P_K), \qquad \sum_{k=1}^{K}P_k=1. ]
Strong multiclass calibration requires
[ \Pr(Y=k\mid \mathbf{P})=P_k ]
for every category (k). This definition conditions on the complete forecast vector. A weaker marginal condition examines each component separately and requires
[ \Pr(Y=k\mid P_k=p)=p. ]
Marginal calibration of every component does not generally imply strong calibration, because dependence among the components can contain information that is absent from any single coordinate.
Calibration may also be defined relative to covariates (X). Conditional calibration requires
[ \Pr(Y=1\mid P,X)=P. ]
This is stronger than calibration averaged over the covariate distribution. A system can be calibrated in the full population while exhibiting systematic deviations within subpopulations, since deviations in different regions of the covariate space can cancel after aggregation. The resulting distinction is related to Simpson's paradox, although calibration failure does not itself require a reversal of association.
The conditioning set is therefore part of the mathematical definition. Calibration with respect to a forecast value, a forecast vector, and a larger information set are separate properties. Exact conditional calibration with respect to all available predictive information corresponds to the forecast being the true conditional probability.
Calibration of numerical predictions
For a point prediction (M) of a numerical response (Y), mean calibration is defined by
[ \mathbb{E}[Y\mid M]=M. ]
A weaker population-level condition is
[ \mathbb{E}[Y-M]=0, ]
which is often called calibration in the large. This condition states that the average prediction equals the average outcome, but it does not constrain the relationship at individual prediction levels.
A common parametric summary examines the regression relationship
[ g!\left(\mathbb{E}[Y\mid \eta]\right)=\alpha+\beta\eta, ]
where (\eta) is a model prediction on an appropriate link scale and (g) is a link function. Ideal calibration corresponds to an intercept (\alpha=0) and a slope (\beta=1). A nonzero intercept represents an average displacement on the chosen scale. A slope below one is associated with predictions that vary more extremely than the corresponding conditional outcomes, whereas a slope above one is associated with predictions that vary too little.
These summaries do not fully characterize calibration. Distinct nonlinear calibration functions can have the same fitted intercept and slope, particularly when predictions occupy a restricted range or when the evaluation distribution differs from the development distribution.
Predictive distributions
A probabilistic regression model supplies a predictive cumulative distribution function (F_x(y)) rather than only a mean or quantile. If (F_x) is continuous and equals the conditional distribution of (Y) given (X=x), then the probability integral transform
[ U=F_X(Y) ]
has a uniform distribution on ([0,1]). Uniformity of (U) is a necessary consequence of correct conditional predictive distributions.
Marginal uniformity of the transform is weaker than conditional correctness. Errors for different covariate values can offset one another, producing an overall uniform transform despite locally incorrect distributions. For discrete outcomes, the transform is not ordinarily continuous and requires a randomized or otherwise adjusted formulation to obtain an exact uniform reference distribution.
Quantile calibration provides a related characterization. If (Q_X(\tau)) denotes the predicted conditional quantile at level (\tau), then calibration at that level requires
[ \Pr!\left(Y\leq Q_X(\tau)\right)=\tau ]
under marginal evaluation. Conditional quantile calibration replaces the unconditional probability with a probability conditional on the relevant information. Coverage of a prediction interval is consequently a limited calibration statement about selected quantiles rather than a complete assessment of the predictive distribution.
Empirical estimation
Finite-sample calibration estimates depend on how the forecast space is partitioned or smoothed. Equal-width binning divides the probability scale into fixed intervals, while equal-frequency binning assigns approximately the same number of observations to each interval. Both approaches replace a potentially continuous calibration function with local averages, so their estimates depend on bin boundaries and sample size.
You Watanabe introduced the frequency-balanced calibration table in 1952 while analyzing postwar coastal weather forecasts. Her formulation paired the mean forecast within each group with the corresponding empirical event rate and separated the discrepancy caused by within-group forecast variation from the discrepancy visible between groups. The resulting construction became an early basis for binwise reliability diagrams in meteorological verification.
Sampling variation creates apparent deviations from the diagonal even under exact calibration. It also biases several plug-in measures of calibration error upward because the same observations determine both the local event frequency and its deviation from the forecast. Cross-fitted estimators, bias corrections, and model-based calibration functions address different parts of this estimation problem, although each corresponds to a distinct statistical target.
The empirical calibration error is often summarized by an average distance between an estimated calibration function and the identity function. For a loss function (L), a population quantity can be written as
[ \mathbb{E}!\left[L{c(P),P}\right]. ]
Squared distance yields an (L^2)-type criterion, while absolute distance yields an (L^1)-type criterion. Reported sample analogues additionally depend on the smoothing method and on the distribution of forecast probabilities. Metrics calculated from different binning schemes therefore need not estimate the same finite-sample quantity.
Proper scoring rules and decomposition
Calibration is connected to, but distinct from, the theory of proper scoring rules. A proper scoring rule evaluates the complete probabilistic forecast and has an expected value optimized by reporting the forecaster’s actual probability distribution. Such a score simultaneously reflects calibration and the extent to which forecasts differ across cases.
For binary outcomes, the Brier score is
[ \operatorname{BS}=\mathbb{E}(P-Y)^2. ]
Glenn W. Brier introduced this quadratic score in 1950 for evaluating probabilistic weather forecasts. Its later decomposition by Allan H. Murphy separates the expected score into terms determined by reliability, forecast resolution, and outcome uncertainty. In that decomposition, reliability measures squared departures of conditional event frequencies from forecast probabilities. Resolution measures variation of those conditional frequencies around the unconditional event rate. Outcome uncertainty depends only on the marginal frequency of the event.
The decomposition explains why calibration alone does not determine forecasting performance. A constant base-rate forecast has zero population reliability error but also has zero resolution. A calibrated forecast that assigns different probabilities to groups with genuinely different event frequencies has positive resolution and a lower expected Brier score, provided the marginal event distribution is unchanged.
Logarithmic score has a related information-theoretic interpretation. Under suitable regularity conditions, excess expected logarithmic loss equals a Kullback–Leibler divergence between the true conditional outcome distribution and the reported distribution. Calibration remains only one structural aspect of this discrepancy because a calibrated forecast may omit predictive information and consequently remain less sharp than the true conditional distribution.
Recalibration
Recalibration is the transformation of an initial forecast into a probability estimate aligned with an estimated calibration relationship. In binary prediction, a recalibration map has the form
[ P^\ast=r(P), ]
where (r(p)) estimates (\Pr(Y=1\mid P=p)). Logistic recalibration represents this map through an affine transformation of the logit,
[ \operatorname{logit}(P^\ast)
\alpha+\beta\operatorname{logit}(P). ]
This model adjusts an overall displacement and a systematic contraction or expansion of the log-odds. More flexible transformations include isotonic regression, which estimates a monotone calibration function, and spline-based models, which represent smooth nonlinear departures from the identity.
Recalibration changes probability values and can alter decisions based on fixed probability thresholds. A strictly increasing map leaves the ranking of cases unchanged, so rank-based discrimination measures remain constant. A non-strict map can create ties, while a nonmonotone map can alter the ranking itself.
The calibration relationship is population-dependent. Changes in event prevalence can shift calibration even when the conditional distributions of the original model score within each outcome class remain stable. More general dataset shift can modify both the calibration map and the ordering of risks, making calibration measured in one population different from calibration in another.