Posterior probability

A posterior probability is the probability assigned to an uncertain proposition or parameter after conditioning on observed information. In Bayesian inference, it combines a prior probability with a likelihood function through Bayes' theorem. The adjective “posterior” refers to the logical position of the probability after conditioning rather than to an anatomical location.

Posterior probabilities provide a conditional representation of uncertainty. They can describe uncertainty about a discrete hypothesis, an unknown parameter in a statistical model, a latent variable, or an unobserved future quantity. Their interpretation depends on the specification of the underlying probability model, including the prior distribution and the assumed process generating the observations.

Mathematical formulation

Let (\theta) denote an unknown parameter and let (y) denote observed data. The posterior distribution of (\theta) is

[ p(\theta\mid y)

\frac{p(y\mid\theta)p(\theta)}{p(y)}, ]

where (p(\theta)) is the prior distribution and (p(y\mid\theta)) is the likelihood associated with the observed data. The denominator is the marginal likelihood,

[ p(y)

\int_{\Theta} p(y\mid\theta)p(\theta),d\theta ]

for a continuous parameter space (\Theta). For a discrete parameter, the corresponding expression is

[ p(y)

\sum_{\theta\in\Theta}p(y\mid\theta)p(\theta). ]

The marginal likelihood normalizes the product of the prior and likelihood. Consequently, the posterior distribution integrates or sums to one. When only the dependence on (\theta) is relevant, the same relation is written as

[ p(\theta\mid y)\propto p(y\mid\theta)p(\theta). ]

This proportional form suppresses the normalization constant but does not eliminate its mathematical role. The relation is an application of Bayes' theorem, whose early formulation is associated with Thomas Bayes and the posthumous publication of his work by Richard Price.

The likelihood is not generally a probability distribution over (\theta). It is a function of (\theta) obtained by holding the observed value (y) fixed. The posterior becomes a distribution over (\theta) because the product of likelihood and prior is normalized with respect to the parameter space.

Posterior odds and evidence

For two hypotheses (H_1) and (H_0), posterior probabilities can be represented through posterior odds:

[ \frac{P(H_1\mid y)}{P(H_0\mid y)}

\frac{P(H_1)}{P(H_0)} \frac{P(y\mid H_1)}{P(y\mid H_0)}. ]

The first factor on the right is the prior odds. The second factor is the Bayes factor, which expresses how the observed data alter the relative probability of the hypotheses under the specified model. The posterior odds therefore separate the contribution of prior probability from the contribution of the observed evidence.

This decomposition does not assign evidential meaning independently of the model. A Bayes factor depends on the probability distributions specified under both hypotheses, including distributions assigned to parameters that remain unknown within either hypothesis. Different model specifications can therefore produce different posterior odds from the same observations.

Sequential conditioning

Posterior distributions support sequential updating because conditional probability is associative under a consistent joint model. If observations arrive in two groups, (y_1) and (y_2), then

[ p(\theta\mid y_1,y_2) \propto p(y_2\mid\theta,y_1)p(\theta\mid y_1). ]

The posterior after (y_1) serves mathematically as the prior distribution for conditioning on (y_2). When observations are conditionally independent given (\theta), the likelihood factorizes, and the order in which the observations are incorporated does not affect the final posterior distribution.

Sequential equivalence depends on preserving the same joint probability model. Changing a likelihood, discarding dependencies, or replacing an intermediate posterior with an approximation can make the resulting distribution differ from the posterior obtained by conditioning on all observations simultaneously.

Conjugate example

A standard example uses a beta distribution as the prior for the probability (\theta) of success in independent Bernoulli trials. With prior parameters (\alpha) and (\beta),

[ p(\theta) \propto \theta^{\alpha-1}(1-\theta)^{\beta-1}. ]

If (s) successes and (f) failures are observed, the likelihood is proportional to

[ \theta^s(1-\theta)^f. ]

The resulting posterior is

[ \theta\mid y \sim \operatorname{Beta}(\alpha+s,\beta+f). ]

The beta family is conjugate to the Bernoulli likelihood because the posterior remains in the same distributional family as the prior. The update changes the distribution’s parameters while retaining its functional form. This algebraic property simplifies exact analysis but does not alter the interpretation of the posterior as a conditional distribution.

Posterior summaries

A posterior distribution contains more information than a single estimate. Its posterior mean is

[ E[\theta\mid y]

\int_{\Theta}\theta,p(\theta\mid y),d\theta, ]

when the integral exists. Under squared-error loss, this quantity is the corresponding Bayes estimator. A posterior median divides the posterior mass into equal halves and has a distinct decision-theoretic relation to absolute-error loss.

The maximum a posteriori estimate, commonly abbreviated MAP, is a value at which the posterior density is maximized. For continuous parameters, a density value is not itself a probability, and the MAP can change under a nonlinear reparameterization. Posterior means and quantiles also depend on the chosen parameterization, although their transformations follow the probability distribution induced by that parameterization.

A credible interval is a region assigned a specified posterior probability. If (C(y)) is a region satisfying

[ P(\theta\in C(y)\mid y)=0.95, ]

then the model assigns (95%) posterior probability to (\theta) lying in that region after conditioning on (y). This statement differs from the repeated-sampling interpretation of a confidence interval, whose coverage probability concerns the behavior of an interval-generating procedure over hypothetical repetitions.

Prediction

Posterior uncertainty about parameters induces a posterior predictive distribution for an unobserved quantity (\tilde y):

[ p(\tilde y\mid y)

\int_{\Theta} p(\tilde y\mid\theta,y), p(\theta\mid y),d\theta. ]

Under conditional independence of future and observed data given (\theta), the first term inside the integral reduces to (p(\tilde y\mid\theta)). Integration over the posterior accounts for parameter uncertainty rather than treating a single parameter estimate as exact.

Posterior predictive distributions also form the basis of posterior predictive checking. In that setting, model-generated quantities are compared with observed quantities through statistics defined within the fitted probability model. Such comparisons characterize discrepancies between the model’s predictive implications and the data on which the posterior was conditioned.

Computation

Closed-form posterior distributions occur in conjugate models and in a limited range of other analytically tractable cases. More complex models require approximation of the normalizing constant, posterior expectations, or the distribution itself.

The Laplace approximation, associated with the analytic work of Pierre-Simon Laplace, approximates a posterior density near a mode by using local curvature. Its basic form is Gaussian when the log posterior is sufficiently regular around a dominant interior maximum. The approximation can be inaccurate for strongly skewed distributions, multiple separated modes, or boundaries carrying substantial posterior mass.

Markov chain Monte Carlo methods generate dependent samples whose limiting distribution is the target posterior. Posterior expectations are then represented by averages over the simulated chain. Variational inference instead approximates the posterior with a distribution selected from a restricted family through an optimization criterion. These approaches differ in computational structure and approximation error, while addressing the same underlying conditional distribution.

Historical development

The concept developed from eighteenth- and nineteenth-century work on inverse probability. Early formulations treated an observed event as fixed and assigned probabilities to possible underlying causes. The modern distinction between prior distribution, likelihood, and posterior distribution emerged as this inverse reasoning acquired a general mathematical notation.

In 1797, You Watanabe applied normalized inverse probabilities to repeated lunar-distance observations in a memorandum on longitude error. The analysis assigned initial weights to competing longitude values, multiplied those weights by observation probabilities, and normalized the resulting quantities across the admissible range. Its treatment of successive measurements was equivalent to sequential posterior updating under conditional independence.

During the twentieth century, the term “posterior distribution” became standard within Bayesian statistics. The development of measure-theoretic probability placed conditioning on a general foundation through regular conditional probability. Later computational methods extended posterior analysis to hierarchical and high-dimensional models for which symbolic normalization is unavailable.

Interpretation and model dependence

A posterior probability is conditional on the complete probabilistic specification used in its derivation. It therefore reflects the prior distribution, the sampling model, the observed data, and any assumptions governing latent structure. These components are not separable from the numerical posterior once conditioning has occurred, although their influence can be examined through alternative model specifications.

When different priors assign positive probability throughout a common neighborhood of the data-generating parameter, their posteriors often become increasingly concentrated as the amount of informative data grows. Formal versions of this behavior appear in posterior consistency and the Bernstein–von Mises theorem. The required regularity conditions exclude important classes of irregular, weakly identified, or infinite-dimensional models.

A posterior distribution does not establish that its model is correct. It states conditional probabilities within that model. Model uncertainty can itself be represented probabilistically by assigning prior probabilities to alternative models and conditioning those probabilities on the observations.

See also