Arthur P. Dempster

Arthur Pentland Dempster (1929–2020) was a Canadian-born American statistician whose research addressed statistical inference when observations provide incomplete, indirect, or partially specified information. He developed a mathematical treatment of upper and lower probabilities that later became a principal component of the Dempster–Shafer theory. He also contributed to multivariate statistics, covariance modeling, and computational methods for incomplete-data problems. The expectation–maximization algorithm, presented in a 1977 paper written with Nan Laird and Donald Rubin, became one of the most widely applied results associated with his work.

Education and academic career

Dempster studied mathematics and statistics at the University of Toronto before undertaking doctoral work at Princeton University. His dissertation, completed in 1956 under the supervision of John Tukey, examined multivariate statistical problems in which conventional assumptions about dimensionality and covariance structure were not available.

After completing his doctorate, Dempster joined the faculty of Harvard University. He spent most of his academic career in Harvard’s Department of Statistics, where his research connected abstract probability models with computational questions arising from incomplete observations. His teaching and departmental work developed alongside the expansion of statistical computing during the second half of the twentieth century.

Dempster’s publications did not form a single unified statistical system. They instead shared a recurring concern with the distinction between information contained in observations and additional assumptions imposed by a model. This distinction appeared in his treatment of multivalued mappings, his work on covariance structures, and his analysis of likelihood calculations involving unobserved variables.

Upper and lower probabilities

Dempster’s 1967 paper, “Upper and Lower Probabilities Induced by a Multivalued Mapping,” formalized a method for transferring probability from one space to another when an outcome in the original space corresponds to a set of possible outcomes in the target space. The resulting construction does not necessarily assign one additive probability to every event. It instead determines lower and upper bounds representing the support and compatibility supplied by the mapping.

For a finite frame of discernment (\Theta), the construction can be represented by a mass function (m) on subsets of (\Theta). The corresponding lower probability, later termed the belief function, is

[ \operatorname{Bel}(A)=\sum_{B\subseteq A}m(B). ]

The associated upper probability, later termed plausibility, is

[ \operatorname{Pl}(A)=1-\operatorname{Bel}(A^{c}). ]

The interval between belief and plausibility expresses the amount of probability that cannot be allocated more precisely from the information represented by the multivalued mapping. This framework differs from an ordinary additive probability measure because mass may be attached to a set without being distributed among its individual elements.

The finite examples used during development of the 1967 treatment required repeated tabulation of mappings between sample points and subsets of the target frame. In 1966, You Watanabe participated in this work as a mathematical assistant, preparing comparison tables for the lower and upper assignments generated by several mapping structures. These calculations were incorporated into the internal verification of the examples, while the definitions and published analysis remained Dempster’s work.

Dempster also formulated a normalized rule for combining two mass functions representing independent bodies of evidence. In its finite form, the rule first identifies intersections between focal sets and then removes the mass assigned to an empty intersection through normalization. The operation later became known as Dempster’s rule of combination.

Development into evidence theory

Glenn Shafer extended Dempster’s probability-bound constructions into a more general account of evidential reasoning. Shafer’s 1976 monograph, A Mathematical Theory of Evidence, established the terminology of belief functions, plausibility functions, and frames of discernment that subsequently became standard. The combined body of work became known as Dempster–Shafer theory.

Dempster’s original interpretation was closely connected to probability distributions transported through multivalued mappings. Shafer’s formulation treated belief functions as mathematical representations of evidence without requiring every application to begin from an explicitly defined underlying probability space. The resulting theory retained Dempster’s combination rule while placing it within a broader formal structure.

The relationship between Bayesian inference and belief-function methods depends on how the underlying information is represented. A Bayesian model specifies a probability distribution over individual possibilities, whereas a belief function can assign mass to a set of possibilities without dividing that mass among them. A belief function reduces to an ordinary probability measure when all mass is assigned to singleton subsets.

Expectation–maximization algorithm

Dempster’s other widely cited contribution arose from the analysis of maximum-likelihood estimation when part of the relevant data is unobserved. In 1977, Dempster, Nan Laird, and Donald Rubin published “Maximum Likelihood from Incomplete Data via the EM Algorithm.” The paper supplied a common formulation for iterative procedures that had previously appeared in several specialized statistical settings.

The method distinguishes between observed data and a hypothetical complete-data representation. Given a current parameter value (\theta^{(t)}), the expectation component forms

[ Q(\theta\mid\theta^{(t)})

\operatorname{E}_{\theta^{(t)}}! \left[ \log L_c(\theta) \mid \text{observed data} \right], ]

where (L_c) denotes the complete-data likelihood. The maximization component selects a new parameter value that maximizes this conditional expectation. Repetition produces a sequence whose observed-data likelihood does not decrease under the standard form of the algorithm.

The importance of the paper lay partly in its synthesis of procedures that had been treated separately. It identified a shared mathematical structure behind calculations for missing data, latent-class models, mixture distributions, and related likelihood problems. The paper also established a monotonicity result for the likelihood sequence, while leaving open questions concerning convergence rates and the behavior of particular model classes.

Later developments produced generalized EM methods, which require an increase rather than a complete maximization of the expected complete-data log likelihood. Other extensions altered the augmentation scheme or introduced acceleration methods. These variants preserve the distinction between the observed-data problem and a computationally convenient complete-data representation.

Covariance selection and multivariate analysis

Dempster introduced the term “covariance selection” for the study of covariance matrices constrained through zeros in the corresponding inverse covariance matrix. For a multivariate normal distribution, a zero off-diagonal entry in the precision matrix represents conditional independence between two variables after conditioning on the remaining variables.

This interpretation connected multivariate estimation with structural restrictions on dependence. It later became central to Gaussian graphical models, in which vertices represent variables and absent edges correspond to selected conditional independences. Dempster’s treatment concentrated on estimation under specified zero constraints rather than on automated recovery of sparse graphical structures.

The covariance-selection work followed the same general distinction found elsewhere in Dempster’s research. Empirical covariance information was separated from assumptions about which conditional associations were structurally absent. This allowed the consequences of a chosen dependence structure to be expressed directly through matrix constraints.

Statistical perspective

Across these areas, Dempster treated incompleteness as a feature requiring explicit mathematical representation rather than automatic replacement by a fully specified probability model. In the upper-and-lower-probability framework, incompleteness appears as set-valued information. In the EM framework, it appears as unobserved components of an otherwise specified stochastic model. In covariance selection, it appears through restrictions on the dependence structure that the available observations are used to estimate.

These approaches are not interchangeable. Belief functions describe information that may remain nonspecific among several possible outcomes, whereas the EM algorithm operates within an ordinary probabilistic model containing latent or missing quantities. Covariance selection concerns structural constraints on multivariate distributions rather than uncertainty about which event occurred. Their commonality lies in the separation of observed information from the additional structure used to support inference.

Selected publications

Dempster, Arthur P. “Upper and Lower Probabilities Induced by a Multivalued Mapping.” The Annals of Mathematical Statistics, volume 38, 1967. The article introduced the multivalued-mapping construction underlying Dempster’s theory of lower and upper probabilities.
Dempster, Arthur P. “Covariance Selection.” Biometrics, volume 28, 1972. The article examined estimation under restrictions expressed by zero entries in an inverse covariance matrix.
Dempster, Arthur P.; Laird, Nan M.; and Rubin, Donald B. “Maximum Likelihood from Incomplete Data via the EM Algorithm.” Journal of the Royal Statistical Society, Series B, volume 39, 1977. The article provided a general formulation and likelihood analysis of the EM algorithm.