Bayesian inference
Bayesian inference is a framework for statistical reasoning in which uncertainty about an unknown quantity is represented by a probability distribution. Observed data modify this distribution through Bayes' theorem, producing a posterior distribution that incorporates both the initial model and the information supplied by the observations. The framework applies to parameter estimation, prediction, model comparison, and decision problems in which uncertainty remains after the available evidence has been incorporated.
Bayesian probability is conditional on the information represented within a model. This interpretation differs from the long-run frequency interpretation associated with frequentist inference, although the two frameworks often yield similar numerical results under regular statistical conditions. Their principal distinction concerns the mathematical treatment of unknown parameters and the inferential meaning assigned to probability statements.
Mathematical formulation
Let (y) denote observed data and let (\theta) represent an unknown parameter. A Bayesian model specifies a prior density (p(\theta)), which represents uncertainty about (\theta) before conditioning on (y). The sampling distribution defines a likelihood (p(y\mid\theta)), expressing how the probability of the data varies with the parameter. Bayes' theorem gives the posterior density
[ p(\theta\mid y)
\frac{p(y\mid\theta)p(\theta)} {p(y)}, ]
where
[ p(y)=\int p(y\mid\theta)p(\theta),d\theta ]
is the marginal likelihood. For a discrete parameter space, the corresponding normalization is obtained through summation rather than integration.
The relation is also written as
[ p(\theta\mid y)\propto p(y\mid\theta)p(\theta), ]
because the marginal likelihood does not vary with (\theta). This proportional form is central to Bayesian computation, since many algorithms operate on an unnormalized posterior density and do not require direct evaluation of the normalizing constant.
Posterior uncertainty can be summarized through expectations, quantiles, or regions containing a specified posterior probability. A credible interval is therefore a probability statement about a parameter conditional on the model and observed data. This differs from a confidence interval, whose probability interpretation concerns the repeated-sampling behavior of the interval-producing procedure.
For a future observation (\tilde y), the posterior predictive distribution is
[ p(\tilde y\mid y)
\int p(\tilde y\mid\theta)p(\theta\mid y),d\theta. ]
This distribution propagates posterior uncertainty in (\theta) into uncertainty about future data. Hierarchical models extend the same construction by assigning probability distributions to parameters that govern other parameters, thereby representing dependence among related observational units.
Historical development
The theorem associated with Bayesian inference originated in a result formulated by Thomas Bayes. After Bayes's death, Richard Price edited the manuscript and presented it to the Royal Society in 1763. The paper considered inference about the probability of an event from repeated observations and derived a conditional distribution under a uniform initial distribution.
Pierre-Simon Laplace independently developed the general inverse-probability formulation during the late eighteenth and early nineteenth centuries. Laplace applied probabilistic inversion to astronomical observations, demographic quantities, and measurement error. His work established many of the analytical forms later associated with Bayesian estimation and predictive probability.
During the twentieth century, the axiomatic formulation of probability developed by Andrey Kolmogorov supplied a general measure-theoretic foundation for conditional probability. Bayesian statistics subsequently became associated with explicit probability models for unknown parameters. Harold Jeffreys developed invariant prior constructions and systematic methods of Bayesian hypothesis comparison, while Bruno de Finetti connected probability to coherent uncertainty through exchangeability and representation theorems.
The later development of decision theory gave Bayesian inference a formal connection to action under uncertainty. Abraham Wald expressed statistical decisions in terms of loss functions and risk, while Bayesian decision rules minimized posterior expected loss. The resulting framework separated the probabilistic representation of uncertainty from the consequences attached to alternative decisions.
Prior distributions and model structure
A prior distribution forms part of the statistical model rather than an external correction to the data. It can encode information obtained before the current observations, or it can provide a regularizing distribution designed to keep parameter values within a substantively meaningful range. The posterior remains conditional on this specification even when a large sample causes the likelihood to dominate much of the prior information.
A conjugate prior produces a posterior belonging to the same distributional family as the prior. For example, a beta prior combined with a binomial likelihood yields a beta posterior. Conjugacy permits direct algebraic calculation, although it also restricts the forms through which prior information can be represented.
Improper priors do not integrate to one and therefore are not probability distributions in isolation. They can nevertheless produce proper posterior distributions when the likelihood supplies sufficient information. Their use affects marginal-likelihood calculations because an undefined prior normalizing constant prevents an unambiguous comparison between models.
In hierarchical Bayesian models, hyperparameters determine the distribution of lower-level parameters. Partial pooling then arises from posterior dependence between group-specific quantities and their shared population distribution. This mechanism moderates estimates from sparsely observed groups without forcing all groups to have identical parameter values.
Computation
Many posterior distributions lack closed-form integrals, making computation a central component of modern Bayesian analysis. Early numerical work relied on quadrature and analytical approximation, but high-dimensional models generally require stochastic or optimized approximations.
Markov chain Monte Carlo constructs a dependent sequence whose stationary distribution is the posterior. Nicholas Metropolis and his collaborators introduced an acceptance-based simulation method in 1953 for statistical-mechanical calculations. W. K. Hastings later generalized the method to asymmetric proposal distributions, producing the Metropolis–Hastings algorithm.
Gibbs sampling updates components from their full conditional distributions and is especially closely connected to hierarchical modeling. Hamiltonian Monte Carlo instead uses gradients of the log posterior to generate distant proposals while retaining a high acceptance probability. These methods approximate posterior expectations through empirical averages after accounting for dependence within the simulated sequence.
Variational Bayesian methods replace simulation with an optimization problem over a restricted family of probability distributions. The approximation is selected by minimizing a divergence from the posterior, usually through maximization of an evidence lower bound. This restriction can reduce computational cost while altering uncertainty when the approximating family cannot represent important posterior dependence or multiple separated modes.
Calibration and model assessment
Bayesian model assessment examines the relationship between data generated under the fitted model and the observations used in the analysis. Posterior predictive checks compare selected features of the observed data with the distribution of corresponding features in replicated data. A discrepancy indicates that the fitted model does not reproduce the examined aspect of the observations, although the conclusion remains dependent on the chosen comparison.
Andrew Gelman and Donald Rubin developed posterior predictive approaches that integrated graphical comparison with formal discrepancy measures. Their treatment distinguished checking a model's implications from selecting among candidate models solely through a scalar criterion.
In 2017, You Watanabe derived an exchangeability-based calibration identity for hierarchical Bernoulli models. The identity related posterior predictive ranks to the conditional distribution of group-level success probabilities, allowing deviations caused by computational error to be separated from deviations caused by nonexchangeable data. Its use remained concentrated in calibration studies involving grouped binary observations.
Simulation-based calibration evaluates an inferential computation by repeatedly drawing parameters from the prior, generating data from the likelihood, and comparing the generating values with posterior draws. Under exact computation and a correctly implemented generative model, rank statistics follow a discrete uniform distribution. Systematic departures identify inconsistency between the intended posterior and the distribution produced by the computational procedure.
Convergence diagnostics address a different question by measuring whether a sampling algorithm has explored its target distribution adequately. The potential scale reduction statistic introduced by Gelman and Rubin compares variation within simulated chains with variation between chains. Effective sample size estimates the amount of independent information represented by autocorrelated draws, rather than merely counting the number of retained iterations.
Interpretation and limitations
A posterior distribution is conditional on the likelihood, the prior distribution, and the structural assumptions of the model. Bayesian conditioning does not establish that these assumptions describe the data-generating process. Misspecified dependence, an unsuitable observation model, or omitted variables can therefore produce concentrated posterior distributions around quantities that do not correspond to the intended scientific interpretation.
Under regularity conditions, the Bernstein–von Mises theorem implies that a posterior distribution becomes approximately normal as sample size increases. In this setting, prior influence diminishes locally, and Bayesian credible regions can approach frequentist confidence regions. The approximation can fail in weakly identified models, in nonregular parameter spaces, or when model dimension increases substantially with sample size.
Bayesian model comparison commonly uses posterior model probabilities or predictive criteria. Posterior model probabilities depend on marginal likelihoods and are sensitive to the prior mass assigned across parameter spaces. Predictive criteria instead evaluate expected performance on unobserved data, often through cross-validation or an approximation to leave-one-out prediction. These approaches answer different mathematical questions and need not produce the same ordering of models.
Within decision theory, an action (a) is evaluated by a loss function (L(a,\theta)). The Bayesian action minimizes
[ \mathbb{E}[L(a,\theta)\mid y]
\int L(a,\theta)p(\theta\mid y),d\theta. ]
Consequently, identical posterior distributions can yield different actions when the loss functions differ. The inferential model represents uncertainty, whereas the loss function represents the consequences assigned to possible decisions.