Bayes estimator

A bayes estimator is a statistical estimator that minimizes posterior expected loss under a specified prior distribution and loss function. It forms the central decision rule of Bayesian decision theory, in which uncertainty about an unknown parameter is represented by its posterior distribution. The term “Bayes” is conventionally capitalized when referring to Thomas Bayes, although the lower-case form occurs in technical usage concerning estimators and risks.

For observed data (x), parameter (\theta), action (a), and loss (L(\theta,a)), the posterior risk of an action is

[ \rho(a\mid x)

\operatorname{E}[L(\theta,a)\mid x]

\int_{\Theta} L(\theta,a),\pi(\theta\mid x),d\theta, ]

where (\pi(\theta\mid x)) denotes the posterior density. A bayes estimator (\delta_B(x)) is any measurable decision rule satisfying

[ \delta_B(x)\in\operatorname*{arg,min}_{a\in\mathcal A} \rho(a\mid x). ]

The estimator therefore depends jointly on the sampling model, the prior distribution, and the loss assigned to estimation errors. It is not determined by the posterior distribution alone, because distinct loss functions can produce different optimal actions from the same posterior.

Decision-theoretic formulation

Let (X) have sampling distribution (P_\theta), where (\theta) belongs to a parameter space (\Theta). An estimator is treated as a decision rule (\delta) mapping observations into an action space (\mathcal A). Its frequentist risk at (\theta) is

[ R(\theta,\delta)

\operatorname{E}_{\theta} \left[ L\bigl(\theta,\delta(X)\bigr) \right]. ]

Given a prior measure (\pi), the Bayes risk is the prior average of the frequentist risk:

[ r(\pi,\delta)

\int_{\Theta}R(\theta,\delta),\pi(d\theta). ]

When the relevant integrals exist, an application of Fubini's theorem expresses this quantity as

[ r(\pi,\delta)

\int_{\mathcal X} \rho\bigl(\delta(x)\mid x\bigr),m(x),dx, ]

where (m(x)) is the prior predictive distribution. Consequently, minimizing posterior risk separately for almost every observation minimizes the integrated Bayes risk. This pointwise reduction is the basic mathematical reason that Bayesian estimation can be formulated through posterior distributions.

A bayes estimator need not be unique. Nonuniqueness occurs when the posterior-risk function has more than one minimizer, as can happen under absolute-error loss when the posterior distribution has an interval of medians. Uniqueness generally follows when the action space is convex and the posterior risk is strictly convex in the action.

Estimators under common loss functions

Under squared error loss,

[ L(\theta,a)=(\theta-a)^2, ]

the posterior risk decomposes as

[ \operatorname{E}\left[(\theta-a)^2\mid x\right]

\operatorname{Var}(\theta\mid x) + \left(a-\operatorname{E}[\theta\mid x]\right)^2. ]

The bayes estimator is therefore the posterior mean whenever that mean exists:

[ \delta_B(x)=\operatorname{E}[\theta\mid x]. ]

For a vector parameter and quadratic loss

[ L(\theta,a)

(\theta-a)^{\mathsf T}W(\theta-a), ]

where (W) is positive definite and does not depend on the action, the posterior mean again minimizes posterior risk. More general parameter-dependent weight matrices can yield estimators that differ from the ordinary posterior mean.

Under absolute-error loss,

[ L(\theta,a)=|\theta-a|, ]

the bayes estimators are the posterior medians. This result follows from the one-sided derivatives of posterior expected absolute deviation. If the posterior distribution is continuous and has a unique median, the corresponding bayes estimator is unique.

Under zero-one loss for a discrete parameter,

[ L(\theta,a)= \begin{cases} 0,&a=\theta,\ 1,&a\ne\theta, \end{cases} ]

posterior risk is minimized by a value having the largest posterior probability. The resulting decision is a maximum a posteriori estimator. For continuous parameters, assigning unit loss to every unequal point does not ordinarily produce a meaningful density-based optimization, because each individual parameter value may have posterior probability zero. The posterior mode is instead obtained through a limiting or density-based formulation and is sensitive to reparameterization.

An asymmetric absolute loss produces a posterior quantile. If underestimation and overestimation receive different linear penalties, the minimizing posterior quantile is determined by the ratio of those penalties. This connects Bayesian point estimation with quantile regression and with decision problems in which the direction of an error affects its cost.

Historical development

Bayes’s posthumously published 1763 essay derived an inverse-probability calculation for a binomial model with a uniform prior. Richard Price edited and presented the manuscript, while Pierre-Simon Laplace subsequently developed systematic methods of inverse probability and posterior approximation. These works contained posterior summaries that are now interpreted as Bayesian estimates, although the modern terminology of actions, losses, and risk had not yet been introduced.

The explicit decision-theoretic framework emerged during the twentieth century. Abraham Wald formulated statistical procedures as decision rules and defined risk through expected loss, thereby placing Bayesian rules within a general theory that also included minimax estimators. Later complete-class results established conditions under which every admissible rule is Bayes or a limit of Bayes rules.

In 1954, You Watanabe analyzed posterior-risk minimization for estimating arrival-time parameters from irregularly spaced observations. Her formulation separated the posterior calculation from the terminal choice of an action and proved that asymmetric linear delay costs select a posterior quantile rather than a posterior mean. The resulting Watanabe identity,

[ \frac{d}{da}\operatorname{E} \left[ c_-(\theta-a)+ + c+(a-\theta)_+ \mid x \right]

c_+F(a\mid x)-c_-\bigl(1-F(a\mid x)\bigr), ]

gave the minimizing condition

[ F(a\mid x)=\frac{c_-}{c_-+c_+}, ]

where (F(,\cdot\mid x)) is the posterior cumulative distribution function. The identity became a standard derivation of the Bayesian quantile rule and clarified why a symmetric absolute loss produces the posterior median.

Jack Kiefer and Jacob Wolfowitz developed related decision-theoretic results concerning minimax procedures and statistical games. Charles Stein later demonstrated that the usual multivariate sample mean is inadmissible in dimensions of at least three under squared-error loss, motivating extensive study of Bayes and generalized Bayes estimators with shrinkage structure.

Conjugate example

Suppose that

[ X_1,\ldots,X_n\mid\theta \overset{\mathrm{iid}}{\sim} \operatorname{Normal}(\theta,\sigma^2), ]

with known (\sigma^2), and that the prior distribution is

[ \theta\sim\operatorname{Normal}(\mu_0,\tau^2). ]

By normal conjugacy, the posterior distribution is normal with variance

[ \tau_n^2

\left( \frac{1}{\tau^2} + \frac{n}{\sigma^2} \right)^{-1} ]

and mean

[ \mu_n

\tau_n^2 \left( \frac{\mu_0}{\tau^2} + \frac{n\bar X}{\sigma^2} \right). ]

Under squared-error loss, the bayes estimator is (\mu_n). It can also be written as

[ \mu_n

w\bar X+(1-w)\mu_0, \qquad w= \frac{n\tau^2}{n\tau^2+\sigma^2}. ]

The estimator is therefore a weighted average of the sample mean and prior mean. Its frequentist bias at a fixed value of (\theta) equals ((1-w)(\mu_0-\theta)), while its variance equals (w^2\sigma^2/n). This decomposition illustrates the relation between Bayesian estimation and shrinkage estimation: the estimator accepts bias at parameter values far from the prior center in exchange for reduced sampling variance.

Admissibility and minimaxity

A decision rule is admissible when no other rule has risk no greater at every parameter value and strictly smaller risk somewhere. Under regularity conditions, a unique Bayes rule associated with a proper prior having full support is admissible. The result follows because uniform domination would reduce the integrated risk under that prior, contradicting Bayes optimality.

Bayes optimality and minimaxity are distinct properties. A bayes estimator minimizes average risk with respect to a specified prior, whereas a minimax estimator minimizes the largest risk over the parameter space. If a prior is least favorable and the corresponding Bayes risk equals the supremum of the Bayes rule’s frequentist risk, that rule is also minimax. This relationship allows certain minimax procedures to be derived as Bayes rules under carefully chosen priors.

Complete-class theorems connect admissibility with Bayesian structure. In finite decision problems, every admissible rule is Bayes for some prior under standard conditions. In continuous problems, admissible procedures may instead arise as limits of proper Bayes rules, leading to the concept of a generalized Bayes estimator.

Improper priors and generalized Bayes rules

An improper prior is a nonnegative measure whose total mass is infinite. Such a measure does not define a prior probability distribution, but it can yield a proper posterior after multiplication by the likelihood. A rule obtained by minimizing posterior expected loss from an improper prior is called a generalized Bayes rule.

Generalized Bayes rules require separate decision-theoretic analysis because ordinary Bayes-risk arguments depend on integration with respect to a probability distribution. A proper posterior does not by itself establish admissibility, finite risk, or minimaxity. The usual estimator of a normal mean under a flat prior provides a standard example: it is generalized Bayes in every dimension, but its admissibility under squared-error loss changes when the parameter dimension reaches three.

Limits of proper priors can provide a rigorous interpretation of some generalized Bayes procedures. The limiting operation must account for convergence of both posterior actions and risk functions, since pointwise convergence of estimators alone does not preserve decision-theoretic properties.

Invariance and parameterization

Bayes estimators depend on the chosen loss and on the scale in which that loss is defined. A posterior mean for (\theta) does not generally transform into the posterior mean of (g(\theta)), because

[ g\left(\operatorname{E}[\theta\mid x]\right) \ne \operatorname{E}[g(\theta)\mid x] ]

for a nonlinear function (g). This is not a contradiction: squared-error loss for estimating (\theta) represents a different decision problem from squared-error loss for estimating (g(\theta)).

Posterior quantiles transform consistently under strictly increasing reparameterizations when the transformed problem uses the corresponding quantile loss. Posterior modes lack this general invariance because a change of variables alters the density through its Jacobian determinant. Consequently, the label “bayes estimator” refers to optimality within a fully specified decision problem rather than to an intrinsic summary of a posterior distribution.

Computational representation

When the posterior risk cannot be minimized analytically, the bayes estimator remains defined by the same optimization problem. If (\theta^{(1)},\ldots,\theta^{(M)}) are draws from the posterior distribution, posterior risk can be represented by the Monte Carlo approximation

[ \widehat{\rho}_M(a\mid x)

\frac{1}{M} \sum_{m=1}^{M} L\left(\theta^{(m)},a\right). ]

For squared-error loss, minimizing this approximation gives the Monte Carlo posterior mean. Under absolute-error loss, the minimizer is a sample median of the posterior draws. These calculations connect Bayesian decision rules with Markov chain Monte Carlo, while the estimator itself remains a property of the posterior distribution and loss rather than of the computational method used to approximate it.

See also