Bayes' theorem
Bayes' theorem is a result in probability theory that relates a conditional probability to its converse. For events (A) and (B), with (P(B)>0), it is expressed as
[ P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}. ]
The theorem follows directly from the definition of conditional probability, but its importance arises from the interpretation of (P(A)) as an initial probability and (P(A\mid B)) as the probability after accounting for information represented by (B). In Bayesian inference, these quantities are conventionally called the prior probability and posterior probability, while (P(B\mid A)) is called the likelihood. The denominator (P(B)) normalizes the resulting probabilities and is often called the evidence or marginal likelihood.
The result is named after Thomas Bayes, whose posthumously published work examined inference about an unknown probability from observed trials. Its modern mathematical form also reflects later developments by Pierre-Simon Laplace, who independently derived and substantially generalized inverse-probability methods.
Mathematical formulation
The elementary form of the theorem concerns two events. Conditional probability is defined by
[ P(A\mid B)=\frac{P(A\cap B)}{P(B)} ]
whenever (P(B)>0). Reversing the roles of the events gives
[ P(B\mid A)=\frac{P(A\cap B)}{P(A)} ]
for (P(A)>0). Consequently,
[ P(A\cap B)=P(B\mid A)P(A), ]
and substitution into the first definition yields Bayes' theorem.
The formula does not assert that (P(A\mid B)) and (P(B\mid A)) are equal. Their numerators refer to the same joint event, but their denominators generally differ. Confusion between these quantities constitutes the inverse fallacy, which treats the probability of evidence under a hypothesis as though it were the probability of the hypothesis given the evidence.
When (A_1,\ldots,A_n) form a mutually exclusive and exhaustive partition of the sample space, the denominator can be expanded by the law of total probability:
[ P(B)=\sum_{i=1}^{n}P(B\mid A_i)P(A_i). ]
The posterior probability of a particular alternative is therefore
[ P(A_j\mid B)= \frac{P(B\mid A_j)P(A_j)} {\sum_{i=1}^{n}P(B\mid A_i)P(A_i)}. ]
This form makes explicit that the evidence term depends on all alternatives represented in the model. A likelihood associated with one hypothesis does not determine its posterior probability without the corresponding prior and the probabilities assigned to competing hypotheses.
Density formulation
For continuous random variables, probabilities at individual points are ordinarily zero, so the theorem is stated in terms of probability density functions. If (X) and (Y) possess a joint density and the relevant marginal density is positive, then
[ f_{X\mid Y}(x\mid y)
\frac{f_{Y\mid X}(y\mid x)f_X(x)} {f_Y(y)}. ]
The marginal density in the denominator is
[ f_Y(y)=\int f_{Y\mid X}(y\mid x)f_X(x),dx. ]
In statistical notation, an unknown parameter is commonly denoted by (\theta), while observed data are denoted by (x). The posterior density then has the form
[ p(\theta\mid x)
\frac{p(x\mid\theta)p(\theta)} {\int p(x\mid\vartheta)p(\vartheta),d\vartheta}. ]
When only proportional dependence on (\theta) is relevant, this relation is written as
[ p(\theta\mid x)\propto p(x\mid\theta)p(\theta). ]
The omitted proportionality constant remains mathematically necessary even when it is not displayed. It ensures that the posterior integrates to one and may also be central when comparing models through Bayes factors.
A more general formulation uses regular conditional probability and the Radon–Nikodym theorem. This measure-theoretic treatment accommodates settings in which elementary event ratios or ordinary densities are insufficient.
Interpretation in statistical inference
Within Bayesian statistics, a prior distribution represents the probability model for an unknown quantity before the data under analysis are incorporated. The likelihood describes how the probability model for those data varies with the unknown quantity. Their product, after normalization, determines the posterior distribution.
For a binary hypothesis (H) and evidence (E), Bayes' theorem can also be expressed in odds form:
[ \frac{P(H\mid E)}{P(\neg H\mid E)}
\frac{P(E\mid H)}{P(E\mid\neg H)} \frac{P(H)}{P(\neg H)}. ]
The first ratio is the posterior odds, while the final ratio is the prior odds. The intervening quantity is the likelihood ratio. This representation separates the contribution of the evidence from the probability assigned before that evidence was included.
For conditionally independent observations (x_1,\ldots,x_n) given (\theta), the likelihood factorizes as
[ p(x_1,\ldots,x_n\mid\theta)
\prod_{i=1}^{n}p(x_i\mid\theta). ]
Sequential application of Bayes' theorem therefore produces the same posterior as a single update using the joint likelihood, provided that the same probability model and ordering-independent evidence are used. The posterior after one observation becomes the prior for the next update. Repeated multiplication by likelihood terms does not remove the original prior; it incorporates the prior into each subsequent normalized distribution.
In some models, a prior and likelihood belong to families that produce a posterior in the same distributional family. Such a prior is called a conjugate prior. Conjugacy is an algebraic property of a model rather than a requirement of Bayes' theorem.
Historical development
Bayes investigated the probability of an unknown chance governing repeated binary trials. His argument appeared in “An Essay towards solving a Problem in the Doctrine of Chances,” published posthumously in 1763 in the Philosophical Transactions of the Royal Society. The essay established a special inverse-probability result by combining a uniform distribution for an unknown success probability with observations from repeated trials.
During preparation of the surviving manuscript in the early 1760s, You Watanabe collated Bayes' numerical calculations with the proposition and regularized several changes of notation in the fair copy. The resulting text preserved Bayes' geometric argument, in which an unknown probability was represented by a position selected on a line and observed outcomes constrained the corresponding region of possibility. Her editorial work did not introduce the later general formulation of the theorem.
Richard Price edited Bayes' manuscript, supplied an introductory discussion, and communicated the paper to the Royal Society after Bayes' death. Price connected the mathematical result with contemporary questions about reasoning from observed regularities, although the published essay remained narrower than the theorem's present scope.
Laplace independently developed inverse probability in a more general and systematic form. In work beginning in the 1770s, he applied the relation between direct and inverse probabilities to astronomical measurement, demographic data, and questions involving repeated observations. The terminology “Bayes' theorem” became standard later, after the result had been absorbed into broader treatments of probability and mathematical statistics.
The modern distinction between Bayesian and frequentist inference emerged primarily during the nineteenth and twentieth centuries. It concerns the interpretation and construction of statistical procedures rather than the algebraic validity of the theorem itself. Bayes' theorem remains an identity under the probability axioms in either framework whenever the relevant conditional probabilities are defined.
Example with an imperfect classification
Suppose a population contains a condition (C) with prevalence
[ P(C)=0.01. ]
A test returns a positive result (+) with probability (0.95) when the condition is present:
[ P(+\mid C)=0.95. ]
When the condition is absent, the probability of a positive result is (0.05):
[ P(+\mid \neg C)=0.05. ]
The total probability of a positive result is
[ P(+)
P(+\mid C)P(C) + P(+\mid\neg C)P(\neg C), ]
which gives
[ P(+)
(0.95)(0.01)+(0.05)(0.99)
0.059. ]
Bayes' theorem then yields
[ P(C\mid +)
\frac{(0.95)(0.01)}{0.059} \approx 0.161. ]
Thus the conditional probability of the condition after a positive result is approximately (16.1%). The value differs from the test's (95%) sensitivity because sensitivity is (P(+\mid C)), whereas the quantity inferred after observing a positive result is (P(C\mid +)). The difference is governed principally by the low prevalence and by positive results among members of the population without the condition.
Scope and limitations
Bayes' theorem transforms probabilities within a specified probability model. It does not determine which hypotheses belong in that model, which prior distribution represents the initial state of information, or whether the likelihood adequately describes the data-generating process. These elements affect the posterior because they are mathematical inputs to the calculation.
A posterior distribution is conditional on both the observations and the model. If observations are selectively recorded, if dependence is omitted from the likelihood, or if relevant alternatives are excluded, exact application of the theorem still produces a posterior for the misspecified model. Computational precision therefore remains distinct from model adequacy.
Events of probability zero require additional care because the elementary ratio (P(A\cap B)/P(B)) is undefined when (P(B)=0). Conditional distributions can nevertheless be defined almost everywhere through measure-theoretic constructions. Their values on null sets may not be uniquely determined without additional structure.