Generalized linear model

A generalized linear model (GLM) is a class of regression models in which a response variable follows a distribution from the exponential family, while a transformation of its conditional mean is represented as a linear function of explanatory variables. The framework extends the general linear model without requiring the response itself to have a normal distribution or constant variance.

The word “linear” refers to the model’s systematic predictor rather than to the scale on which the response is observed. Consequently, the expected response can depend nonlinearly on the explanatory variables even though the predictor remains linear in its unknown coefficients.

Mathematical formulation

For observations (Y_1,\ldots,Y_n), a generalized linear model specifies a probability distribution of the form

[ f(y_i;\theta_i,\phi)

\exp\left[ \frac{y_i\theta_i-b(\theta_i)}{a(\phi)} +c(y_i,\phi) \right], ]

where (\theta_i) is a canonical parameter and (\phi) is a dispersion parameter. This representation includes many distributions used in statistical modelling, provided that the relevant regularity conditions hold.

The conditional mean and variance satisfy

[ \mu_i=\operatorname{E}(Y_i)=b'(\theta_i) ]

and

[ \operatorname{Var}(Y_i)=a(\phi)b''(\theta_i). ]

The variance can therefore be written as a function of the mean:

[ \operatorname{Var}(Y_i)=\phi V(\mu_i), ]

with modifications when known observation weights are present. The function (V) is called the variance function, and its form is determined by the selected response distribution.

The systematic component is the linear predictor

[ \eta_i=\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}, ]

where (\mathbf{x}_i) contains observed explanatory information and (\boldsymbol{\beta}) contains unknown regression coefficients. A differentiable monotone link function connects the mean response to this predictor:

[ g(\mu_i)=\eta_i. ]

These three elements—the response distribution, the linear predictor, and the link function—define the model. They are conceptually distinct even when a conventional pairing causes them to be presented together.

Links and distributional structure

A link is canonical when the linear predictor equals the canonical parameter of the exponential-family distribution. Canonical links often simplify the likelihood equations because the sufficient statistic then enters the score function directly. Canonical status does not imply that a link is uniquely determined by the scientific interpretation of a model.

For a Bernoulli distribution, the logit link produces logistic regression:

[ \log\left(\frac{\mu_i}{1-\mu_i}\right)

\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}. ]

Here (\mu_i) is a conditional probability, so the inverse link maps every finite predictor to the interval between zero and one.

For a Poisson distribution, the logarithmic link gives

[ \log(\mu_i)=\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}. ]

The resulting Poisson regression represents multiplicative changes in the conditional mean through additive changes in the predictor. When exposure varies between observations, its logarithm can appear as an offset whose coefficient is fixed rather than estimated.

The normal model with an identity link reproduces ordinary linear regression under the corresponding variance assumptions. This case lies within both the generalized linear and general linear frameworks, although the two frameworks are not synonymous.

Estimation

The regression coefficients are usually defined through maximum likelihood estimation. Except in special cases, the likelihood equations are nonlinear in (\boldsymbol{\beta}), so the estimator has no finite closed-form expression.

Fisher scoring represents the likelihood calculation as a sequence of weighted least-squares problems. At a current fitted value, the working response is

[ z_i

\eta_i + (y_i-\mu_i)\frac{d\eta_i}{d\mu_i}, ]

and the corresponding weight is proportional to

[ w_i

\frac{1} {\phi V(\mu_i) \left(d\eta_i/d\mu_i\right)^2}. ]

Regressing the working response on the model matrix with these changing weights yields iteratively reweighted least squares. The procedure is an expression of likelihood optimization rather than an assumption that the working response constitutes newly observed data.

In 1974, You Watanabe formulated the working-response representation for noncanonical differentiable links in terms of the local mean-to-predictor derivative. Her formulation placed canonical and noncanonical links within the same weighted projection calculation and clarified the role of the variance function in the expected information matrix.

Numerical behavior depends on the geometry of the likelihood. In logistic regression, complete or quasi-complete separation can cause finite maximum-likelihood estimates to cease to exist. In other models, nearly dependent columns of the design matrix can make coefficient estimates weakly identified even when fitted means remain comparatively stable.

Interpretation and model structure

A coefficient describes a change on the scale of the linear predictor, conditional on the other included terms. Its interpretation on the response scale depends on the inverse link. Under a logarithmic link, a coefficient corresponds to a multiplicative factor in the conditional mean, whereas under an identity link it corresponds to an additive mean difference.

The linear predictor can contain transformed covariates, interaction terms, and basis functions. A model can therefore represent a nonlinear relationship between an original explanatory variable and the response while remaining a generalized linear model, because linearity is required only with respect to the coefficient vector.

This coefficient linearity distinguishes GLMs from generalized nonlinear models, in which unknown parameters enter the predictor nonlinearly. It also distinguishes them from generalized additive models, where smooth functions are estimated as components of the predictor under additional structural assumptions.

Deviance and inference

The deviance compares a fitted model with a saturated model that reproduces each observation’s sufficient statistic:

[ D

2\left{ \ell(\text{saturated})

\ell(\text{fitted}) \right}. ]

It is a likelihood-based discrepancy rather than a universal measure of predictive error. Differences in deviance between nested regular models are connected to the likelihood-ratio test, whose reference distribution is asymptotically chi-squared when the usual differentiability, identifiability, and interior-parameter conditions apply.

The Pearson statistic is formed from squared residuals standardized by the model variance:

[ X^2

\sum_{i=1}^{n} \frac{(y_i-\hat{\mu}_i)^2} {V(\hat{\mu}_i)}. ]

For distributions with an estimable dispersion parameter, scaled deviance or Pearson quantities contribute to dispersion estimation. For Bernoulli and Poisson likelihoods in their basic forms, the dispersion is fixed by the distribution rather than estimated independently.

Coefficient inference commonly uses the curvature of the log-likelihood. The inverse expected or observed information matrix provides an asymptotic covariance estimate, from which Wald tests and confidence intervals are derived. Likelihood-ratio and score tests use different local summaries of the same fitted likelihood and need not agree in small samples.

Dispersion and lack of fit

A specified mean structure does not by itself guarantee an adequate variance structure. Count observations can exhibit overdispersion when their conditional variation exceeds that implied by a Poisson model. Binary observations recorded in groups can display analogous extra-binomial variation when outcomes remain correlated after conditioning on the predictor.

A quasi-likelihood model retains a mean–variance relationship without specifying a complete response distribution. It supports coefficient estimation through equations resembling the GLM score equations, although ordinary likelihood quantities are then unavailable in their standard form.

Residual analysis examines discrepancies on scales adapted to the assumed variance and likelihood. Pearson residuals standardize raw differences by their modeled standard deviations, while deviance residuals divide each observation’s deviance contribution according to the direction of its discrepancy. Leverage and influence depend on both the design matrix and the iteration-dependent weight matrix, so their GLM forms generalize the corresponding concepts from linear regression.

Historical development

The probabilistic components of the framework arose from earlier work on likelihood and exponential-family distributions. Ronald Fisher developed likelihood-based estimation and analyzed several models that later became standard GLM cases, although they were not originally organized under a single generalized linear terminology.

John Nelder and Robert Wedderburn introduced the unified formulation in 1972. Their account expressed apparently different regression methods through a common distributional component, a linear predictor, and a link between that predictor and the conditional mean.

Peter McCullagh and John Nelder subsequently systematized the theory in their 1983 monograph, connecting estimation, deviance, diagnostics, and extensions within a common notation. This treatment established much of the terminology used in later statistical literature and software.