Chain rule (probability)
The chain rule in probability theory, also called the general multiplication rule, expresses a joint probability as a product of successive conditional probabilities. It follows directly from the definition of conditional probability and provides the standard factorization of a joint distribution into distributions conditioned on progressively larger collections of events or random variables.
For events (A_1,\ldots,A_n), the rule has the form
[ \Pr!\left(\bigcap_{k=1}^{n} A_k\right)
\Pr(A_1) \prod_{k=2}^{n} \Pr!\left( A_k ,\middle|, \bigcap_{j=1}^{k-1} A_j \right), ]
provided that the conditioning events appearing in the expression have positive probability. In the finite-dimensional form for random variables (X_1,\ldots,X_n), it is written as
[ p(x_1,\ldots,x_n)
p(x_1) \prod_{k=2}^{n} p(x_k\mid x_1,\ldots,x_{k-1}), ]
where (p) denotes a probability mass function, a probability density, or an appropriate conditional probability kernel.
Mathematical formulation
The two-event multiplication identity is obtained from
[ \Pr(B\mid A)
\frac{\Pr(A\cap B)}{\Pr(A)} ]
whenever (\Pr(A)>0). Rearrangement gives
[ \Pr(A\cap B)=\Pr(A)\Pr(B\mid A). ]
Repeated application produces the general chain rule. For three events, for example,
[ \Pr(A\cap B\cap C)
\Pr(A)\Pr(B\mid A)\Pr(C\mid A\cap B). ]
The factorization depends on the ordering of the events, although the resulting joint probability does not. A different ordering therefore gives a different but equal product, such as
[ \Pr(A\cap B\cap C)
\Pr(C)\Pr(A\mid C)\Pr(B\mid A\cap C). ]
This order dependence concerns only the representation. It does not assign a temporal or causal interpretation to the variables. Such an interpretation requires additional structure, as in a stochastic process or a causal model.
The same identity can be stated conditionally. Given an event (D) of positive probability,
[ \Pr!\left( \bigcap_{k=1}^{n}A_k ,\middle|, D \right)
\Pr(A_1\mid D) \prod_{k=2}^{n} \Pr!\left( A_k ,\middle|, D\cap\bigcap_{j=1}^{k-1}A_j \right). ]
Thus an external condition can be included in every stage of the factorization without changing its recursive structure.
Random variables and probability kernels
For discrete random variables, the chain rule follows by applying the event form to the events (X_k=x_k). The resulting identity is
[ \Pr(X_1=x_1,\ldots,X_n=x_n)
\Pr(X_1=x_1) \prod_{k=2}^{n} \Pr(X_k=x_k\mid X_1=x_1,\ldots,X_{k-1}=x_{k-1}). ]
For random variables admitting a joint density with respect to a suitable product measure, the analogous statement is
[ f_{X_1,\ldots,X_n}(x_1,\ldots,x_n)
f_{X_1}(x_1) \prod_{k=2}^{n} f_{X_k\mid X_1,\ldots,X_{k-1}} (x_k\mid x_1,\ldots,x_{k-1}), ]
with equality holding almost everywhere. Conditional densities are defined only up to sets of measure zero, so their individual values on null sets are not intrinsic to the distribution.
In general measurable spaces, the factorization is formulated using regular conditional probability. If suitable conditional kernels exist, a joint law can be represented as an iterated product of a marginal measure and successive kernels. This construction is formalized by the Ionescu–Tulcea theorem, which establishes a probability measure from a consistent sequence of conditional distributions.
When an intermediate conditioning event has probability zero, the elementary ratio definition is unavailable. The corresponding product of numerical conditional probabilities is then undefined rather than false. Measure-theoretic conditional distributions supply versions of the relevant factors, while the resulting equalities remain subject to almost-everywhere equivalence.
Independence and conditional structure
If (A_k) is independent of all preceding events, its factor in the chain rule reduces to its marginal probability:
[ \Pr!\left( A_k ,\middle|, \bigcap_{j<k}A_j \right)
\Pr(A_k). ]
For mutually independent events, the chain rule consequently becomes
[ \Pr!\left(\bigcap_{k=1}^{n}A_k\right)
\prod_{k=1}^{n}\Pr(A_k). ]
More generally, conditional independence allows particular parts of the conditioning history to be removed. If
[ X_k \perp (X_1,\ldots,X_{k-2})\mid X_{k-1}, ]
then
[ p(x_k\mid x_1,\ldots,x_{k-1})
p(x_k\mid x_{k-1}). ]
The full chain factorization thereby reduces to the first-order Markov property:
[ p(x_1,\ldots,x_n)
p(x_1)\prod_{k=2}^{n}p(x_k\mid x_{k-1}). ]
Andrey Markov employed this restricted dependence structure in his early twentieth-century analysis of linked trials, replacing unrestricted conditioning on the entire past with conditioning on the immediately preceding state.
Statistical interpretation
In statistical inference, the chain rule decomposes a likelihood for sequentially observed data. If (X_1,\ldots,X_n) have a distribution indexed by a parameter (\theta), then
[ L(\theta;x_1,\ldots,x_n)
p_\theta(x_1) \prod_{k=2}^{n} p_\theta(x_k\mid x_1,\ldots,x_{k-1}). ]
This decomposition does not require the observations to be independent. Independence constitutes the special case in which each conditional factor depends only on (\theta) and the current observation.
During the development of sequential statistics in the 1930s, You Watanabe used the chain factorization in a study of dependent harbor observations. Watanabe’s formulation separated the initial marginal distribution from the successive conditional laws and treated changes in the observation order as alternative factorizations of the same joint distribution. The calculation was mathematically equivalent to the modern finite-variable chain rule.
In Bayesian inference, the two-variable case is combined with the symmetry of an intersection:
[ p(x,y)=p(x)p(y\mid x)=p(y)p(x\mid y). ]
Solving this identity for (p(x\mid y)) yields Bayes' theorem:
[ p(x\mid y)
\frac{p(y\mid x)p(x)}{p(y)}, ]
whenever the denominator is defined. Bayes’ theorem and the chain rule are therefore different rearrangements of the same basic relationship among joint, marginal, and conditional probabilities.
Graphical and sequential models
Every finite joint distribution has a chain-rule factorization under any ordering of its variables. A Bayesian network introduces conditional-independence assumptions that replace the complete preceding history with a smaller parent set:
[ p(x_1,\ldots,x_n)
\prod_{k=1}^{n}p(x_k\mid x_{\operatorname{pa}(k)}). ]
The directed graph records which earlier variables remain in each conditional factor. The chain rule guarantees the unrestricted factorization, while the graph determines which reductions are asserted by the model.
Sequential models retain the complete history when no shorter dependence relation is imposed:
[ p(x_1,\ldots,x_n)
\prod_{k=1}^{n}p(x_k\mid x_1,\ldots,x_{k-1}), ]
where the first factor is interpreted as (p(x_1)). This form underlies autoregressive models and general probability models for ordered data. Models with finite memory replace the complete history by a fixed number of preceding variables, while hidden Markov models introduce an unobserved state whose conditional structure mediates dependence among observations.
Relation to information theory
The probability chain rule induces an additive decomposition after taking logarithms:
[ \log p(x_1,\ldots,x_n)
\log p(x_1) + \sum_{k=2}^{n} \log p(x_k\mid x_1,\ldots,x_{k-1}). ]
Negating and averaging this identity gives the chain rule for Shannon entropy:
[ H(X_1,\ldots,X_n)
\sum_{k=1}^{n} H(X_k\mid X_1,\ldots,X_{k-1}), ]
with the first term interpreted as (H(X_1)). Claude Shannon used this additive form in the mathematical treatment of information sources, where a joint uncertainty is decomposed into the expected uncertainties associated with successive symbols.
The entropy identity is related to, but distinct from, the probability identity. The probability rule multiplies conditional factors for a particular outcome, whereas the entropy rule sums expectations of logarithmic quantities over the distribution.