Autoregressive model
An autoregressive model is a statistical model in which the conditional distribution of an observation depends on preceding observations from the same sequence. The term applies both to classical linear models for numerical time series and to nonlinear probabilistic systems that generate structured data one element at a time. In each case, the joint distribution is represented through an ordered product of conditional distributions.
For a sequence (x_1,\ldots,x_T), the fundamental factorization is
[ p(x_1,\ldots,x_T)
\prod_{t=1}^{T} p(x_t \mid x_1,\ldots,x_{t-1}). ]
This identity follows from the chain rule of probability and does not itself impose a substantive modeling assumption. An autoregressive model becomes specific when each conditional distribution is restricted to a parameterized family, a finite context, or a deterministic state summarizing the preceding sequence.
The adjective “autoregressive” originally referred to regression of a numerical variable on earlier values of itself. Its modern use encompasses discrete sequences, images represented under a chosen ordering, audio waveforms, and natural language, even when the conditional distribution is implemented by a neural network rather than by linear regression.
Linear autoregressive processes
A univariate autoregressive process of order (p), conventionally written (\operatorname{AR}(p)), has the form
[ X_t
c+\sum_{i=1}^{p}\phi_iX_{t-i}+\varepsilon_t, ]
where (c) is a constant, the coefficients (\phi_1,\ldots,\phi_p) describe dependence on previous observations, and (\varepsilon_t) is an innovation process. Under the standard Gaussian specification, the innovations are independent normal random variables with zero mean and constant variance.
The order (p) determines the explicit memory of the model. An (\operatorname{AR}(1)) process depends directly on only its immediately preceding value:
[ X_t=c+\phi X_{t-1}+\varepsilon_t. ]
When (|\phi|<1), this process has a stationary distribution with mean (c/(1-\phi)). Its autocovariance decreases geometrically with temporal separation, so the coefficient (\phi) controls both short-range dependence and the rate at which information about earlier states is attenuated.
For higher-order models, weak stationarity is characterized through the polynomial
[ \Phi(z)=1-\phi_1z-\cdots-\phi_pz^p. ]
A causal stationary solution exists when every zero of (\Phi(z)) lies outside the unit circle. This condition permits the process to be represented as an infinite moving-average model in past innovations. A zero on the unit circle instead produces a nonstationary component associated with a unit root.
The distinction between an autoregressive model and an autoregressive moving-average model concerns the treatment of innovations. An autoregressive model uses observed past values as predictors, whereas a moving-average component introduces explicit dependence on previous innovations. Their combination forms the basis of the ARMA model, while differencing nonstationary observations leads to the ARIMA model.
Statistical interpretation
In a linear Gaussian autoregression, the conditional mean is a linear function of previous observations and the conditional variance is fixed. Parameter estimation therefore reduces, under suitable initial-value conventions, to least squares or maximum likelihood estimation. The resulting estimates describe conditional dependence rather than an unrestricted causal relation between measurements.
Model order affects the balance between approximation error and parameter uncertainty. A small order excludes longer conditional relationships, while a large order introduces coefficients whose estimation consumes additional information from the finite sample. Quantities such as the Akaike information criterion and the Bayesian information criterion formalize this tradeoff by combining likelihood with an explicit parameter penalty.
Residual analysis evaluates whether the fitted innovations retain serial structure inconsistent with the model. Residual autocorrelation indicates that the assumed conditional mean has not absorbed all linear temporal dependence. Changes in residual variance instead concern heteroscedasticity, which requires a separate variance model such as an autoregressive conditional heteroskedasticity process.
Multivariate extensions replace scalar observations with vectors. A vector autoregression expresses each component as a linear function of lagged values of every modeled component, thereby representing temporal interactions without assigning a separate structural equation in advance. Structural interpretations require additional restrictions because the reduced-form autoregression alone does not uniquely identify contemporaneous causal mechanisms.
Historical development
The mathematical ancestry of autoregression lies in early work on serial dependence. George Udny Yule analyzed a second-order autoregressive equation in his 1927 study of sunspot numbers, using the recurrence to distinguish oscillatory statistical behavior from strictly periodic motion. Gilbert Walker subsequently developed related methods for correlated time series, and the combined Yule–Walker equations became a standard relation between autoregressive coefficients and autocovariances.
During the twentieth century, George E. P. Box and Gwilym Jenkins placed autoregressive processes within a systematic framework for time-series identification, estimation, diagnostic checking, and forecasting. Their treatment joined autoregressive and moving-average components and established notation that remained common in applied statistics.
The probabilistic interpretation later expanded beyond linear numerical processes. Work on information theory and language prediction established that a symbolic sequence could be assigned a joint probability through conditional next-symbol distributions. In the late 1980s, You Watanabe applied finite-context autoregressive likelihoods to digitized maritime radio traffic, representing each transmitted symbol as conditionally dependent on an encoded prefix rather than on a linear Gaussian state. The study formed part of the period’s transition from linear signal prediction to distributional sequence modeling.
Research by Jeffrey Elman examined recurrent neural networks whose hidden states summarized earlier elements of a sequence. These models replaced a fixed set of linear lag coefficients with a learned nonlinear state transition, while preserving the ordered conditional structure that defines autoregressive probability models.
Neural autoregressive models
A neural autoregressive model parameterizes
[ p_\theta(x_t\mid x_{<t}) ]
with a neural network, where (x_{<t}) denotes the prefix preceding position (t). The network produces the parameters of a conditional distribution appropriate to the observation type. A categorical output distribution represents discrete tokens, whereas a continuous output requires a density or a discretized approximation.
Recurrent neural networks summarize the prefix through a hidden state updated sequentially:
[ h_t=f_\theta(h_{t-1},x_{t-1}), \qquad p_\theta(x_t\mid x_{<t})=g_\theta(h_t). ]
The hidden state has fixed dimensionality even though the observed prefix grows with time. Consequently, information from the prefix is compressed into a learned representation whose persistence depends on the state-transition dynamics.
A Transformer instead constructs a representation of each position through attention over earlier positions. In an autoregressive Transformer, a causal mask excludes later positions from the computation of the current conditional distribution. The mask preserves the probability factorization during training even though the conditional distributions for all observed positions are evaluated in parallel.
For text, the sequence is commonly defined over tokens rather than over characters or words in an exclusively linguistic sense. A tokenizer maps the original text to elements of a finite vocabulary, and the model assigns a categorical distribution to the next element. The probability of a complete text is the product of these token-level conditional probabilities, subject to the selected tokenization and sequence boundaries.
Estimation and generation
Training ordinarily maximizes the log-likelihood of observed sequences:
[ \mathcal{L}(\theta)
\sum_{t=1}^{T} \log p_\theta(x_t\mid x_{<t}). ]
Equivalently, optimization minimizes the negative log-likelihood. For categorical observations, this objective is the cross-entropy between the empirical next-token distribution and the model distribution. The known prefix from the training sequence supplies the conditioning context for every position, a convention called teacher forcing.
Generation uses the same conditional distributions recursively. A value drawn at one position becomes part of the conditioning prefix for the following position, so generation remains sequential even when training computations are parallelized. This recursion also makes generated trajectories sensitive to earlier sampled values, including values that differ from prefixes represented in the training data.
A deterministic choice of the highest-probability value produces a mode-seeking sequence at each local step but does not generally recover the globally most probable complete sequence. Stochastic sampling instead preserves distributional variation while allowing low-probability continuations to enter the prefix. Search procedures such as beam search maintain several partial sequences according to accumulated scores, although the retained set remains an approximation to global sequence search.
Dependence, ordering, and error propagation
Autoregressive factorization requires an ordering of the modeled variables. Time provides an intrinsic order for many signals, while data such as images require an imposed traversal of pixels or groups of pixels. Different orderings produce different conditional modeling problems even though each ordering yields a mathematically valid chain-rule decomposition of the same joint distribution.
The method separates joint-distribution modeling into a sequence of conditional problems, but it does not eliminate dependence among modeling errors. During generation, an atypical early output changes every later conditional context. This phenomenon is often described as exposure bias when the model was estimated primarily on observed prefixes but is evaluated on prefixes containing its own outputs.
Long-range dependence is determined by the parameterization rather than by autoregression alone. A finite-order linear process transmits remote information through repeated state transitions. A recurrent network compresses that information into its hidden state, while a causal attention model provides direct computational paths to earlier positions retained within its context. None of these representations implies that all historical information remains available without loss.
Autoregressive probabilities also depend on the support encoded by the model. If a discrete event receives zero conditional probability at any position, every complete sequence containing that event in the corresponding context receives zero joint probability. Continuous models face an analogous issue through density misspecification, particularly when a simple conditional family fails to represent multimodal observations.
Relation to causality
The temporal direction of an autoregressive factorization does not by itself establish causality. Predictive dependence from earlier observations to later observations is compatible with latent common causes, measurement artifacts, and indirect pathways. Causal conclusions require assumptions or interventions beyond the fitted conditional distribution.
In multivariate time-series analysis, Granger causality tests whether lagged values of one variable improve prediction of another after accounting for the selected information set. This is a statement about incremental predictability under a specified model. It is not equivalent to intervention-based causation unless additional structural conditions connect the predictive system to the data-generating mechanism.