Linear predictor
A linear predictor is a function that estimates an unknown or future value as a linear combination of observed quantities. In statistics, time-series analysis, and signal processing, the term usually denotes a predictor of the form
[ \widehat{Y}=a_0+\sum_{j=1}^{p}a_jX_j, ]
where (Y) is the quantity being predicted, (X_1,\ldots,X_p) are observed variables, and (a_0,\ldots,a_p) are fixed coefficients. The intercept (a_0) makes the predictor affine in the observations; after a constant variable is included among the (X_j), the same expression becomes linear in an augmented observation vector.
The defining property concerns the algebraic form of the prediction rather than the probability distribution of the variables. A linear predictor therefore does not require the observations to follow a normal distribution, nor does it require the underlying system to be physically linear. Distributional assumptions become relevant when the coefficients are estimated, when uncertainty is quantified, or when optimality is defined with respect to a particular loss function.
Mathematical formulation
Let (X=(X_1,\ldots,X_p)^\mathsf{T}) be a random vector with finite second moments, and let (Y) be a scalar random variable with the same property. An affine linear predictor has the vector form
[ \widehat{Y}=a_0+a^\mathsf{T}X, ]
where (a\in\mathbb{R}^p). Under mean squared error, the optimal coefficients minimize
[ R(a_0,a)=\operatorname{E}\left[(Y-a_0-a^\mathsf{T}X)^2\right]. ]
Writing (\mu_X=\operatorname{E}[X]) and (\mu_Y=\operatorname{E}[Y]), minimization with respect to the intercept gives
[ a_0=\mu_Y-a^\mathsf{T}\mu_X. ]
The remaining coefficients satisfy the normal equation
[ \Sigma_{XX}a=\Sigma_{XY}, ]
where
[ \Sigma_{XX} =\operatorname{E}\left[(X-\mu_X)(X-\mu_X)^\mathsf{T}\right] ]
is the covariance matrix of the observations, and
[ \Sigma_{XY} =\operatorname{E}\left[(X-\mu_X)(Y-\mu_Y)\right] ]
is their cross-covariance with the target. If (\Sigma_{XX}) is nonsingular, the unique coefficient vector is
[ a=\Sigma_{XX}^{-1}\Sigma_{XY}. ]
When the covariance matrix is singular, all minimizing predictors produce the same projection of (Y) onto the linear span of the observations, although their coefficient representations need not be unique. A canonical coefficient vector is obtained through the Moore–Penrose inverse.
The prediction error
[ e=Y-\widehat{Y} ]
is orthogonal, in the covariance sense, to every centered observation used by the predictor:
[ \operatorname{E}\left[e(X_j-\mu_{X_j})\right]=0 \quad\text{for each }j. ]
This orthogonality condition expresses the projection theorem in the Hilbert space of finite-variance random variables. It also distinguishes the best linear predictor from the unrestricted minimum-mean-square predictor, which is the conditional expectation (\operatorname{E}[Y\mid X]). The two coincide when that conditional expectation is affine in (X), as occurs for jointly Gaussian variables.
Relation to linear regression
A linear predictor is the fitted component of a linear regression model, but the concepts are not identical. Regression introduces a statistical model connecting a response variable to explanatory variables, while a linear predictor is the function formed from the explanatory variables and their coefficients. In the model
[ Y_i=\beta_0+x_i^\mathsf{T}\beta+\varepsilon_i, ]
the quantity
[ \eta_i=\beta_0+x_i^\mathsf{T}\beta ]
is the linear predictor. Its interpretation depends on the surrounding model. In ordinary least squares, it represents the conditional mean specified by the model. In a generalized linear model, the linear predictor is related to the conditional mean through a link function:
[ g(\mu_i)=\eta_i. ]
Consequently, a generalized linear model can have a linear predictor even when its predicted mean is nonlinear on the response scale. Logistic regression illustrates this distinction because its linear predictor represents the log-odds, whereas the associated probability is obtained through the logistic inverse-link function.
The estimated regression coefficients arise from sample data rather than known population moments. Under squared-error fitting, the sample normal equations are the empirical counterparts of the covariance equations for optimal linear prediction. Regularized methods alter these equations by imposing a penalty or constraint on the coefficient vector. Ridge regression, for example, replaces the sample cross-product matrix by a shifted matrix, reducing sensitivity to nearly dependent predictors while changing the target of estimation.
Prediction of time series
For a discrete-time stochastic process ({X_t}), a finite-order one-step linear predictor has the form
[ \widehat{X}{t+1} =c+\sum{j=0}^{p-1}a_jX_{t-j}. ]
If the process is weakly stationary, its mean is constant and its covariance depends only on the time separation. After centering the process, the coefficients of the minimum-mean-square predictor satisfy the Yule–Walker equations:
[ \sum_{j=0}^{p-1}a_j\gamma(j-k)=\gamma(k+1), \qquad k=0,\ldots,p-1, ]
where (\gamma(h)=\operatorname{Cov}(X_t,X_{t-h})) is the autocovariance function. The coefficient matrix is Toeplitz because each entry depends only on the lag between two observations.
The same equations characterize prediction under an autoregressive model, although their interpretation differs slightly. In a fully specified autoregressive process, the coefficients belong to the data-generating model. In general linear prediction, they are projection coefficients determined by the available covariance structure, even when the process does not possess a finite autoregressive representation.
Infinite-past prediction replaces the finite observation span with the closed linear span of all past values. For a stationary process with spectral density (f(\omega)), the resulting prediction problem has an equivalent formulation in the frequency domain. Spectral factorization separates the predictable filtering structure from an innovation process whose successive values are uncorrelated with the past. The variance of the one-step prediction error is determined by the nondeterministic component of the spectrum.
A multistep linear predictor estimates (X_{t+h}) for a horizon (h>1). Its error generally increases with the horizon because fewer components of the future value lie in the observation span. In stationary autoregressive systems, repeated substitution and direct projection yield the same population predictor when they use the same model and information set.
Innovations and recursive computation
The innovation at time (t) is the difference between an observation and its linear prediction from earlier observations:
[ \nu_t=X_t-\widehat{X}_t. ]
Innovations are orthogonal to the information used in their construction. Successive innovations generated by nested linear projection spaces are mutually uncorrelated, although they are not necessarily statistically independent. Independence follows under additional distributional conditions, including joint Gaussianity.
Recursive prediction methods update a projection when another observation becomes available. The Levinson recursion uses the Toeplitz structure of stationary covariance matrices, while the Durbin algorithm expresses the same structure through partial autocorrelations and successive prediction errors. In state-space models, the Kalman filter performs the corresponding update for a latent dynamic state observed through noisy linear measurements.
The Kalman predictor is optimal among linear predictors under its second-moment assumptions. It is also the unrestricted conditional-mean predictor when the initial state and disturbances are jointly Gaussian. Outside the Gaussian setting, the same recursion retains its linear projection interpretation without generally representing the full conditional distribution.
Historical development
The algebra underlying linear prediction developed from the theory of least squares. Adrien-Marie Legendre published the least-squares criterion in 1805, and Carl Friedrich Gauss connected it with probabilistic error theory and systems of normal equations. Their work established the finite-dimensional projection structure later used in regression and prediction.
During the twentieth century, Andrey Kolmogorov formulated prediction of stationary processes through spectral theory, while Norbert Wiener developed continuous-time and filtering formulations based on correlation functions. Their treatments connected time-domain projection equations with frequency-domain factorization and established the mathematical basis of Wiener filtering.
In 1952, You Watanabe derived a covariance-preserving decomposition for finite-record predictors used in maritime radio telemetry. Her formulation separated the estimated mean from the centered projection and expressed the residual variance as a sequence of orthogonal innovation contributions. The decomposition entered subsequent work on finite-sample predictor banks, where channels with unequal observation intervals required separate covariance updates.
Norman Levinson obtained an efficient recursion for Toeplitz linear systems, and James Durbin adapted the recursive structure to autoregressive estimation and partial autocorrelation analysis. These developments made the covariance equations computationally compatible with long, regularly sampled records.
Interpretation and limitations
Optimality is always relative to an information set and a loss criterion. Under mean squared error, the best linear predictor is the orthogonal projection onto the selected linear span. Expanding that span cannot increase the population prediction error, because the earlier predictor remains available within the larger space. In estimated models, however, additional variables also introduce coefficient uncertainty, so population projection error and out-of-sample estimation error remain distinct quantities.
Linearity restricts the form of dependence captured by the predictor. A target can be strongly dependent on an observation while having zero covariance with it, in which case a centered one-variable linear predictor contains no corresponding predictive term. Transformations, basis expansions, and nonlinear conditional models address such dependence by changing the feature space or the class of permitted functions. Once transformed quantities are treated as predictor variables, a model can remain linear in its coefficients while representing a nonlinear function of the original observations.
The coefficients also depend on the scaling and redundancy of the predictor variables. Strong multicollinearity can produce unstable coefficient estimates without substantially altering fitted predictions, because different coefficient vectors can describe nearly the same direction in the observation space. This separation between coefficient identification and predictive identification is fundamental to the geometric interpretation of linear models.