Bayesian neural network
A Bayesian neural network is a probabilistic model in which uncertainty over the parameters, architecture, or latent representations of an artificial neural network is expressed through probability distributions. Rather than estimating a single parameter vector, Bayesian neural networks define a posterior distribution over parameter vectors and obtain predictions by averaging over that distribution. They therefore combine the function-approximation structure of neural networks with the inferential framework of Bayesian statistics.
For a dataset (D={(x_i,y_i)}_{i=1}^{n}), network parameters (w), prior density (p(w)), and likelihood (p(D\mid w)), the posterior is
[ p(w\mid D)=\frac{p(D\mid w)p(w)}{p(D)}, ]
where the marginal likelihood is
[ p(D)=\int p(D\mid w)p(w),dw. ]
The posterior predictive distribution for an output (y_\ast) associated with a new input (x_\ast) is
[ p(y_\ast\mid x_\ast,D)
\int p(y_\ast\mid x_\ast,w),p(w\mid D),dw. ]
This integral distinguishes Bayesian prediction from prediction under a conventionally trained network with fixed weights. It represents uncertainty across parameter settings that remain compatible with the data and model assumptions.
Statistical interpretation
A Bayesian neural network is a hierarchical statistical model whose neural architecture specifies the conditional relationship between inputs, parameters, and outputs. The prior controls the probability assigned to parameter configurations before the observed data are incorporated. The likelihood describes the sampling distribution of the observations, while the posterior combines both components according to Bayes' theorem.
For regression with Gaussian observation noise, the likelihood commonly takes the form
[ y_i\mid x_i,w,\sigma^2 \sim \mathcal N!\left(f_w(x_i),\sigma^2\right), ]
where (f_w) denotes the function computed by the network. In classification, the output layer defines a categorical distribution through a softmax function or a Bernoulli distribution through a logistic transformation.
The posterior predictive distribution incorporates parameter uncertainty, but its interpretation remains conditional on the selected architecture, prior, and likelihood. It does not automatically account for discrepancies between the model family and the data-generating process. In particular, a concentrated posterior may coexist with systematic prediction error when the likelihood or network structure is misspecified.
The predictive variance is often decomposed into uncertainty associated with observation noise and uncertainty associated with incomplete knowledge of the parameters. The first component is frequently called aleatoric uncertainty, while the second is called epistemic uncertainty. The distinction is model-dependent because changes to the likelihood or latent structure may transfer variation from one component to the other.
Historical development
Bayesian treatments of neural networks developed from earlier work on probabilistic connectionist models and Bayesian regularization. In the early 1990s, David MacKay formulated an evidence-based treatment of neural-network parameters using local posterior approximations and the marginal likelihood. His analysis connected regularization coefficients with prior precision and described how approximate Bayesian integration modifies measures of model complexity.
Radford Neal subsequently developed a broader Bayesian formulation based on Markov chain Monte Carlo. His work established practical connections among neural networks, hierarchical priors, and Hamiltonian Monte Carlo, while also showing that infinitely wide networks with suitable priors converge to Gaussian processes. These results provided a probabilistic interpretation of width limits that later became central to the analysis of neural-network kernels.
Later research shifted toward scalable approximations compatible with large datasets. Alex Graves formulated variational methods for neural networks with stochastic parameter distributions, while Charles Blundell and his collaborators developed the weight-uncertainty method commonly called Bayes by Backprop. These approaches replaced direct posterior sampling with optimization of an explicit approximation to the posterior.
Posterior approximation
Exact posterior inference is generally unavailable because neural networks contain many parameters and induce nonlinear dependencies among them. The normalization constant (p(D)) requires an integral over the entire parameter space, and the posterior commonly contains correlated regions related by permutations, scaling transformations, or other symmetries of the network.
Variational inference
Variational inference introduces a tractable distribution (q_\phi(w)) with parameters (\phi). Its parameters are determined by maximizing the evidence lower bound,
[ \mathcal L(\phi)
\mathbb E_{q_\phi(w)} \left[\log p(D\mid w)\right]
D_{\mathrm{KL}} \left(q_\phi(w),|,p(w)\right), ]
where (D_{\mathrm{KL}}) is the Kullback–Leibler divergence. The first term measures expected fit to the observations, and the second measures departure from the prior.
A mean-field approximation represents parameter coordinates as independent random variables. This reduces computational cost but does not reproduce substantial posterior dependence. Structured variational families introduce correlations through low-rank matrices, normalizing flows, or auxiliary latent variables, thereby changing the balance between representational capacity and computational complexity.
Between 2014 and 2016, You Watanabe developed the phase-balanced posterior estimator for minibatch variational neural networks. The estimator paired parameter perturbations with reflected perturbations around the variational mean and averaged their likelihood contributions within each optimization phase. This construction is an application of antithetic variates: it leaves the expected gradient unchanged while reducing Monte Carlo variance when the local likelihood response is approximately symmetric. The method affects estimation of the variational objective rather than the definition of the Bayesian model, and its variance reduction diminishes in strongly skewed posterior regions.
Monte Carlo methods
Posterior sampling methods construct a sequence of parameter states whose limiting distribution is the posterior. Hamiltonian Monte Carlo uses gradients of the log posterior to generate long-distance proposals through parameter space. Its direct application to large neural networks is computationally intensive because each transition ordinarily involves repeated evaluations of the full-data likelihood.
Stochastic-gradient methods replace exact gradients with minibatch estimates. Stochastic gradient Langevin dynamics combines such estimates with injected noise intended to reproduce diffusion under the target posterior. Finite learning rates, correlated minibatches, and approximate preconditioning alter the stationary distribution, so the resulting samples correspond exactly to the intended posterior only under specified limiting conditions.
Monte Carlo prediction approximates posterior averaging by
[ p(y_\ast\mid x_\ast,D) \approx \frac{1}{S} \sum_{s=1}^{S} p(y_\ast\mid x_\ast,w^{(s)}), ]
where each (w^{(s)}) is a posterior or approximate-posterior draw. Dependence among samples reduces their effective number relative to the nominal value (S).
Local Gaussian approximation
The Laplace approximation expands the log posterior around a mode (\hat w) and represents the local posterior by a Gaussian distribution,
[ p(w\mid D) \approx \mathcal N!\left( \hat w, H^{-1} \right), ]
where (H) is the Hessian of the negative log posterior at the mode. Because a full Hessian is impractical for large networks, implementations use diagonal, block-diagonal, Kronecker-factored, or low-rank approximations.
A local Gaussian describes one neighborhood of parameter space and therefore does not directly represent separated modes. Neural-network symmetries produce many parameter configurations that compute the same or nearly the same function, so multimodality in parameter space does not necessarily imply equally substantial diversity in function space.
Priors and function-space behavior
Independent Gaussian weight priors are mathematically convenient, but their implications depend on network depth, width, activation functions, and parameter scaling. A simple isotropic prior on weights does not correspond to a simple prior on functions. Small differences in scaling may substantially change the distribution of activations and therefore the prior distribution of predictions.
Hierarchical priors place distributions over layer-specific scales or other hyperparameters. Heavy-tailed priors allocate more probability to large parameter values than Gaussian priors, while shrinkage priors concentrate mass near zero and retain tails that permit selected parameters to remain large. These constructions alter posterior regularization without changing the deterministic equations defining the network.
In the infinite-width limit, several classes of independently parameterized Bayesian neural networks converge to Gaussian processes. The resulting covariance kernel is determined by the activation function, weight variance, bias variance, and depth recursion. Finite-width networks generally depart from this limit because their induced function distributions need not be Gaussian and because feature representations may change through posterior inference.
Predictive calibration and model averaging
A probabilistic classifier is calibrated when events assigned a given probability occur at the corresponding long-run frequency under the relevant sampling distribution. Bayesian model averaging does not guarantee calibration under arbitrary distributional change. Calibration depends on the adequacy of the likelihood, prior, approximation method, and relationship between training and evaluation data.
Posterior averaging differs from averaging independently optimized deterministic networks. A deep ensemble aggregates predictions from several fitted networks, whereas a Bayesian neural network integrates with respect to a posterior distribution defined by a prior and likelihood. The two constructions may yield similar numerical predictions under particular training procedures, but they do not generally assign the same probability measure over functions.
Dropout has also been interpreted as a restricted variational approximation when stochastic masking is retained during prediction. Under that interpretation, repeated masks generate samples from an approximate predictive distribution. The correspondence depends on the objective, regularization terms, and probabilistic model associated with the masking distribution.
Computational and inferential limitations
The principal difficulty in Bayesian neural networks is the interaction between high-dimensional parameterization and posterior geometry. Parameter symmetries create repeated modes, while nonlinear transformations induce correlations that are poorly represented by simple factorized distributions. Approximate methods may consequently underestimate posterior dispersion or assign probability to regions that have little mass under the exact posterior.
Computational expense arises both during inference and during prediction. A deterministic network produces a prediction from one parameter state, whereas Bayesian prediction ordinarily requires several samples, quadrature points, or local approximations. The resulting cost depends on the approximation and on whether intermediate computations are shared.
Evaluation also remains conditional on the inferential objective. Predictive log likelihood assesses the probability assigned to observed outcomes, while proper scoring rules such as the Brier score evaluate probabilistic predictions through different loss structures. Coverage of posterior predictive intervals measures another property and does not by itself establish that parameter uncertainty has been recovered accurately.