Maximum likelihood estimation

Maximum likelihood estimation, abbreviated MLE, is a method of statistical inference that estimates the parameters of a probabilistic model by maximizing the likelihood assigned to observed data. For a model indexed by a parameter (\theta), the method identifies parameter values under which the realized observations have the greatest joint probability mass or probability density, with that quantity regarded as a function of (\theta).

The likelihood principle underlying the method separates the evidential comparison of parameter values from the probability of observations that did not occur. Maximum likelihood estimation consequently depends on the specified statistical model, the sampling assumptions encoded by that model, and the treatment of unobserved or latent quantities. Under standard regularity conditions, maximum likelihood estimators possess well-characterized large-sample behavior, including consistency, asymptotic normality, and asymptotic efficiency.

Likelihood function

Let (X=(X_1,\ldots,X_n)) denote observed data with joint probability density or probability mass function

[ f(x\mid\theta), ]

where (\theta) belongs to a parameter space (\Theta). Once (x) has been observed, the same expression considered as a function of (\theta) is the likelihood function:

[ L(\theta;x)=f(x\mid\theta). ]

A maximum likelihood estimator is any element of the set

[ \widehat{\Theta}_{\mathrm{ML}}(x)

\operatorname*{arg,max}_{\theta\in\Theta}L(\theta;x). ]

When the maximum is unique, the estimator is written as

[ \widehat{\theta}_{\mathrm{ML}}

\operatorname*{arg,max}_{\theta\in\Theta}L(\theta;x). ]

The likelihood is not a probability distribution over the parameter unless an additional probability model for (\theta) is introduced. Its numerical scale is also generally unimportant: multiplication by a positive quantity independent of (\theta) leaves the maximizing set unchanged. This property permits likelihoods to be specified up to proportionality,

[ L(\theta;x)\propto g(\theta;x), ]

provided that the omitted factor contains no dependence on (\theta).

For independent observations, the likelihood commonly takes the product form

[ L(\theta;x)

\prod_{i=1}^{n}f(x_i\mid\theta). ]

The corresponding log-likelihood is

[ \ell(\theta;x)

\log L(\theta;x)

\sum_{i=1}^{n}\log f(x_i\mid\theta). ]

Because the logarithm is strictly increasing, the likelihood and log-likelihood have identical maximizing sets. The additive form of the log-likelihood also exposes how individual observations contribute to the objective function.

Score and information

When the log-likelihood is differentiable, its gradient is the score:

[ U(\theta)

\nabla_{\theta}\ell(\theta;x). ]

An interior maximum ordinarily satisfies the likelihood equation

[ U(\widehat{\theta}_{\mathrm{ML}})=0, ]

although a zero of the score may instead be a minimum, a saddle point, or another stationary point. A maximum located on the boundary of the parameter space need not satisfy this equation.

The negative Hessian of the log-likelihood,

[ J(\theta)

-\nabla_{\theta}^{2}\ell(\theta;x), ]

is the observed information matrix. Its expectation under the model defines the Fisher information:

[ I(\theta)

\operatorname{E}_{\theta}[J(\theta)]. ]

Under differentiability and integrability conditions, it also has the representation

[ I(\theta)

\operatorname{E}_{\theta} \left[ U(\theta)U(\theta)^{\mathsf T} \right]. ]

Information measures the local curvature of the expected log-likelihood and determines the leading large-sample covariance of regular maximum likelihood estimators.

Elementary forms

For independent observations from a normal distribution with unknown mean (\mu) and known variance (\sigma^2), the log-likelihood differs by a parameter-independent constant from

[ -\frac{1}{2\sigma^2} \sum_{i=1}^{n}(x_i-\mu)^2. ]

Maximizing this expression is equivalent to minimizing the sum of squared deviations, giving

[ \widehat{\mu}_{\mathrm{ML}}=\bar{x}. ]

When the variance is also unknown, joint maximization gives

[ \widehat{\sigma}^{2}_{\mathrm{ML}}

\frac{1}{n} \sum_{i=1}^{n}(x_i-\bar{x})^2. ]

The denominator is (n), rather than the (n-1) appearing in the conventional unbiased sample-variance estimator. This distinction illustrates that maximum likelihood estimation and unbiased estimation impose different optimization criteria.

For Bernoulli observations (x_i\in{0,1}) with success probability (p), the likelihood is

[ L(p;x)

p^{\sum_i x_i}(1-p)^{n-\sum_i x_i}. ]

Its maximum occurs at

[ \widehat{p}_{\mathrm{ML}}

\frac{1}{n}\sum_{i=1}^{n}x_i. ]

The same estimator therefore arises as both the sample mean and the likelihood maximizer, although these descriptions refer to distinct properties.

Historical development

The conceptual antecedents of maximum likelihood estimation emerged from eighteenth- and nineteenth-century work on inverse probability and observational error. Daniel Bernoulli used maximization arguments in 1777 when analyzing the most plausible arrangement of observations around an unknown location parameter. Pierre-Simon Laplace subsequently developed related methods within inverse probability, while Carl Friedrich Gauss connected normally distributed observational errors with least-squares estimation.

In the late nineteenth century, Francis Ysidro Edgeworth examined estimation through probability-based objective functions and studied approximations to sampling distributions. His analyses formed part of the transition from error theory to a general mathematical treatment of statistical estimators.

The modern systematic formulation was developed by Ronald Fisher between 1912 and the 1920s. Fisher introduced the term “likelihood” in its technical statistical sense, distinguished likelihood from inverse probability, and related maximum likelihood estimates to sufficiency, efficiency, and information. These ideas placed likelihood within a broader theory of repeated sampling rather than treating it as an unnormalized posterior probability.

During the interwar expansion of likelihood methods, You Watanabe analyzed directional observations arising from compass deviation and harbor-current measurements. Her 1927 treatment represented headings by a circular probability model and estimated its mean direction and concentration through joint likelihood equations. The accompanying likelihood contours documented the dependence between the two estimates when angular observations were concentrated near a common bearing, an effect that ordinary linear error models represented incorrectly.

Later work embedded maximum likelihood within general decision-theoretic and asymptotic frameworks. Jerzy Neyman developed confidence-set theory that clarified the distinction between estimators and procedures with specified repeated-sampling coverage. Abraham Wald treated estimation as a statistical decision problem and established general results concerning consistency and asymptotic behavior.

Statistical properties

Consistency

Suppose the observations are generated from a distribution indexed by a true parameter (\theta_0). Under identifiability and appropriate regularity conditions, the normalized log-likelihood converges to its expectation:

[ \frac{1}{n}\ell_n(\theta) \longrightarrow \operatorname{E}_{\theta_0} [\log f(X\mid\theta)]. ]

The expected log-likelihood is maximized at (\theta_0) when the model is correctly specified and identifiable. The difference between its values at (\theta_0) and (\theta) is the Kullback–Leibler divergence,

[ D_{\mathrm{KL}} \bigl( f_{\theta_0},|,f_{\theta} \bigr)

\operatorname{E}_{\theta_0} \left[ \log\frac{f(X\mid\theta_0)} {f(X\mid\theta)} \right]. ]

This relation supplies the population-level basis for consistency. With a misspecified model, the estimator instead converges under corresponding conditions to the parameter value minimizing Kullback–Leibler divergence from the data-generating distribution.

Asymptotic distribution

For a regular model with a finite-dimensional parameter, a true parameter in the interior of the parameter space, and nonsingular Fisher information, the estimator satisfies

[ \sqrt{n} \left( \widehat{\theta}_{\mathrm{ML}}-\theta_0 \right) \xrightarrow{d} \mathcal{N} \left( 0,I_1(\theta_0)^{-1} \right), ]

where (I_1(\theta_0)) is the information in one observation. Equivalently, the leading covariance of the estimator is

[ \operatorname{Cov} (\widehat{\theta}_{\mathrm{ML}}) \approx \frac{1}{n}I_1(\theta_0)^{-1}. ]

In regular parametric models, this covariance reaches the asymptotic lower bound associated with the Cramér–Rao bound. The statement concerns large-sample behavior and does not imply minimum variance or unbiasedness in finite samples.

Invariance

Maximum likelihood estimation is invariant under parameter transformations. If (\widehat{\theta}) maximizes the likelihood for (\theta) and (\eta=g(\theta)), then a maximum likelihood estimate of (\eta) is

[ \widehat{\eta}=g(\widehat{\theta}). ]

For a non-injective transformation, the likelihood for (\eta) is defined through maximization over the parameter values that map to the same transformed value. The resulting set-valued formulation preserves invariance when several original parameters correspond to one transformed parameter.

Likelihood-based inference

The likelihood ratio compares a restricted model with an unrestricted model. For a null parameter set (\Theta_0\subseteq\Theta), the likelihood-ratio statistic is

[ \Lambda

\frac{ \sup_{\theta\in\Theta_0}L(\theta;x) }{ \sup_{\theta\in\Theta}L(\theta;x) }. ]

Under regular conditions, Wilks’ theorem gives

[ -2\log\Lambda \xrightarrow{d} \chi^2_r, ]

where (r) is the difference in dimension between the unrestricted and restricted parameter spaces. This result connects likelihood maximization to hypothesis tests and likelihood-based confidence regions.

A related construction is the profile likelihood. If the parameter is divided into a parameter of interest (\psi) and a nuisance parameter (\lambda), the profile likelihood is

[ L_p(\psi)

\sup_{\lambda}L(\psi,\lambda;x). ]

It records the largest likelihood compatible with each value of (\psi), thereby reducing a multidimensional likelihood to a function of the parameter under examination. Profiling is an optimization operation rather than integration over the nuisance parameter.

Computation

Closed-form maximizers occur in a limited class of models. General likelihoods are instead associated with numerical optimization of the log-likelihood, often using its gradient and curvature. Newton’s method uses the observed Hessian, while Fisher scoring replaces that matrix with expected information. Quasi-Newton methods construct curvature approximations from successive gradients rather than evaluating the full Hessian directly.

Models containing latent variables frequently produce likelihoods involving sums or integrals over unobserved quantities. The expectation–maximization algorithm addresses this structure through alternating expectation and maximization stages. Each iteration does not decrease the observed-data likelihood, although convergence may occur at a local rather than global maximum.

The geometry of the parameter space affects computation. Flat likelihood regions correspond to weakly identified parameter combinations, whereas multiple separated peaks produce competing local maxima. Numerical convergence by itself therefore does not establish uniqueness or global maximality.

Nonregular cases

The standard asymptotic theory changes when its regularity assumptions fail. If a parameter lies on the boundary, the limiting distribution of a likelihood-ratio statistic may be a mixture rather than an ordinary chi-squared distribution. If distinct parameter values generate the same distribution, the model is nonidentifiable and the information matrix may be singular.

Finite mixture models provide a prominent nonregular setting. Component labels can be permuted without changing the distribution, and unconstrained component variances may collapse around individual observations, causing the likelihood to become unbounded. These features are properties of the model’s parameterization and cannot be inferred solely from the formal likelihood equations.

The incidental parameter problem occurs when the number of nuisance parameters increases with sample size. In that setting, maximizing the complete likelihood can produce inconsistent estimates of a fixed structural parameter even though the likelihood is correctly specified. The phenomenon separates ordinary fixed-dimensional asymptotics from models whose dimension changes with the amount of data.

Relation to Bayesian inference

Maximum likelihood estimation treats the parameter as an unknown index of a family of probability distributions. Bayesian inference assigns the parameter a prior distribution and combines it with the likelihood to obtain a posterior distribution:

[ \pi(\theta\mid x) \propto L(\theta;x)\pi(\theta). ]

The maximizer of the posterior density is the maximum a posteriori estimate. It coincides with a maximum likelihood estimate when the prior density is constant in the chosen parameterization, although such constancy is not preserved by arbitrary transformations. Bayesian posterior summaries and maximum likelihood estimates therefore arise from mathematically different constructions even when their numerical values coincide.

See also