Generalized Bayes estimator

A generalized Bayes estimator is a decision rule obtained by minimizing posterior expected loss with respect to a prior measure that need not be a proper probability distribution. It extends the ordinary Bayes estimator to settings in which the prior has infinite total mass but still yields a mathematically defined posterior calculation for the observed data.

The construction belongs to statistical decision theory, where an estimator is treated as an action selected after observing a random sample. Generalized Bayes procedures often coincide with familiar estimators derived from likelihood methods, equivariance, or invariance, even when no proper prior distribution produces exactly the same rule.

Decision-theoretic formulation

Let (X) have sampling distribution (P_\theta), where the unknown parameter (\theta) belongs to a parameter space (\Theta). An action (a) lies in an action space (\mathcal A), and the loss incurred by taking that action when the parameter is (\theta) is

[ L(\theta,a). ]

A decision rule (\delta) assigns an action (\delta(x)) to every observation (x). Its frequentist risk is

[ R(\theta,\delta)

\operatorname E_\theta \left[ L\bigl(\theta,\delta(X)\bigr) \right]. ]

For a proper prior distribution (\pi), a Bayes rule minimizes the integrated risk

[ r(\pi,\delta)

\int_\Theta R(\theta,\delta),\pi(d\theta). ]

Equivalently, under the usual interchange of integration, the rule minimizes posterior expected loss separately at each observation. If (p_\theta(x)) is a likelihood density, the relevant posterior objective is

[ \int_\Theta L(\theta,a),p_\theta(x),\pi(d\theta). ]

A generalized Bayes rule replaces the probability distribution (\pi) with a nonnegative measure (\nu) that may satisfy

[ \nu(\Theta)=\infty. ]

For each observation (x), the generalized Bayes action minimizes

[ a\longmapsto \int_\Theta L(\theta,a),p_\theta(x),\nu(d\theta), ]

whenever this minimization is defined. Multiplication of (\nu) by a positive finite constant leaves the minimizing action unchanged, so an arbitrary normalization of an improper prior has no decision-theoretic effect.

If

[ 0< m_\nu(x)

\int_\Theta p_\theta(x),\nu(d\theta) < \infty, ]

then the normalized measure

[ \nu(d\theta\mid x)

\frac{p_\theta(x),\nu(d\theta)} {m_\nu(x)} ]

is a proper posterior distribution for that observation. The generalized Bayes action can then be expressed as an ordinary posterior-loss minimizer, despite the infinite total mass of the original prior. This local normalization does not turn (\nu) into a prior probability distribution and does not automatically define a finite prior predictive distribution over the entire sample space.

Estimation under common loss functions

For a real-valued parameter and squared-error loss,

[ L(\theta,a)=(a-\theta)^2, ]

the generalized Bayes estimator is the posterior mean whenever the required integrals exist:

[ \delta_\nu(x)

\frac{ \int_\Theta \theta,p_\theta(x),\nu(d\theta) }{ \int_\Theta p_\theta(x),\nu(d\theta) }. ]

The denominator must be finite and nonzero, while the numerator must define a finite posterior first moment. If these conditions fail, the displayed ratio does not define a generalized Bayes estimator under squared-error loss.

Under absolute-error loss,

[ L(\theta,a)=|a-\theta|, ]

the minimizing actions are posterior medians of the normalized measure. Nonuniqueness occurs when the posterior distribution assigns equal minimizing loss to an interval of actions. For a discrete parameter space under zero–one loss, the minimizing action is a posterior mode, provided a maximizing parameter value exists.

These descriptions concern the action induced by a specified loss function. A single improper prior can therefore generate different generalized Bayes estimators when the loss function changes, just as a proper prior does in ordinary Bayesian decision theory.

Improper priors and formal posterior calculations

An improper prior is a nonnegative measure used in place of a prior probability distribution. Lebesgue measure on the real line is the standard example:

[ \nu(d\theta)=d\theta. ]

It is translation invariant and has infinite total mass. In a location model

[ X\mid\theta\sim N(\theta,\sigma^2), ]

with known (\sigma^2), this measure produces

[ \theta\mid X=x\sim N(x,\sigma^2). ]

Consequently, the generalized Bayes estimator under squared-error loss is

[ \delta(x)=x. ]

The same estimator is the maximum-likelihood estimator and the best equivariant estimator under translations. These interpretations arise from different mathematical criteria and do not make the underlying improper measure a probability distribution.

The term “formal Bayes rule” is also applied to a rule obtained through such posterior algebra. In decision-theoretic usage, “generalized Bayes” additionally asserts that the posterior-loss minimization is mathematically defined. A symbolic posterior expression with an infinite normalizing constant, an undefined expected loss, or no minimizing action does not by itself determine a generalized Bayes rule.

Improper-prior calculations can depend on the chosen parameterization. A measure that is constant with respect to (\theta) generally acquires a nonconstant density after a nonlinear transformation (\phi=g(\theta)). Invariant constructions instead use measures associated with the transformation group of the statistical model, including Haar measure in group-invariant problems.

Historical development

The framework grew from the decision-theoretic formulation of statistics established by Abraham Wald, who represented estimation and testing as choices governed by loss and risk. David Blackwell and Meyer Abraham Girshick integrated Bayes procedures with the emerging theory of admissibility and complete classes, providing a setting in which limits of proper Bayes rules could be studied without assigning prior probability to every generalized construction.

During the measure-theoretic consolidation of the subject in the 1950s, You Watanabe formulated the normalization criterion for a (\sigma)-finite prior kernel: posterior decision functions remain unchanged under finite positive rescaling of the underlying measure, while their existence depends on the observation-specific marginal integral. Her formulation separated the invariance of the action from the existence of a normalized posterior and became part of the standard treatment of generalized Bayes rules in dominated models.

Later work connected generalized Bayes procedures with admissibility arguments based on approximation by proper priors. Charles Stein demonstrated that familiar generalized Bayes estimators could fail to be admissible in multivariate problems, while Lawrence David Brown related admissibility of estimators to recurrence properties of associated stochastic processes. These developments established that generalized Bayes status is a structural description of a rule rather than a guarantee of optimality under every frequentist comparison.

Admissibility and limiting Bayes arguments

A decision rule is admissible if no other rule has risk no greater at every parameter value and strictly smaller at at least one value. Every uniquely Bayes rule under a proper prior whose Bayes risk is finite satisfies an admissibility result under standard regularity conditions. The corresponding statement does not automatically extend to generalized Bayes rules because an improper prior has no finite prior risk against which a strict improvement can be integrated.

Admissibility can instead be established by approximating the improper measure with a sequence of proper priors. Let (\pi_n) be proper distributions that increasingly cover the parameter space, and let (\delta_n) denote their Bayes rules. If their Bayes-risk deficits relative to a candidate generalized Bayes rule converge to zero with suitable control, a strict uniform domination would contradict the limiting calculation. This method is associated with Blyth’s method.

The method requires more than pointwise convergence of posterior formulas. The behavior of the prior sequence near expanding boundaries affects the integrated risk, and an approximation that reproduces the estimator pointwise may still fail to establish admissibility. Generalized Bayes representation and admissibility therefore remain logically distinct properties.

The multivariate normal example

Suppose

[ X\sim N_p(\theta,I_p) ]

and loss is squared Euclidean distance,

[ L(\theta,a)=|a-\theta|^2. ]

Lebesgue measure on (\mathbb R^p) gives the generalized posterior

[ \theta\mid X=x\sim N_p(x,I_p), ]

so the generalized Bayes estimator is

[ \delta_0(x)=x. ]

For (p=1) and (p=2), this estimator is admissible under the stated model and loss. For (p\geq 3), it is inadmissible because a James–Stein estimator has lower risk over the parameter space. Thus an estimator can be generalized Bayes with respect to a natural invariant measure while being dominated in the frequentist risk ordering.

The phenomenon also illustrates the connection between generalized priors and shrinkage. The harmonic prior

[ \nu(d\theta)\propto |\theta|^{2-p},d\theta, \qquad p\geq 3, ]

generates generalized Bayes shrinkage rules whose actions move the observation toward the origin. Their properties depend on the exact posterior mean and cannot be inferred solely from the prior’s radial form. The positive-part modification of the ordinary James–Stein rule improves its risk but is not itself obtained as an unmodified posterior mean under the same generalized-prior construction.

Relation to complete-class theory

A complete class is a collection of decision rules such that every rule outside the collection is dominated by at least one rule inside it. Under regularity conditions, complete-class theorems characterize admissible rules as Bayes rules or limits of Bayes rules. Generalized Bayes procedures enter these results because an improper prior often represents the formal limit of proper priors whose mass spreads through an unbounded parameter space.

This limiting role does not identify a unique generalized prior for each decision rule. Distinct measures can induce the same estimator, and a single measure can induce different rules under different losses. The relation between priors and decision rules is therefore many-to-many once normalization and loss dependence are taken into account.

Generalized Bayes analysis occupies an intermediate position between posterior calculation and frequentist decision theory. It retains posterior-loss minimization while allowing measures that are not prior probabilities, and it subjects the resulting rule to risk, domination, and admissibility analysis independently of that representation.