Generalized linear mixed model

A generalized linear mixed model (GLMM) is a statistical model that combines the response-distribution framework of a generalized linear model with the latent random effects of a mixed model. It represents responses whose conditional distributions belong to an exponential family, while allowing observations within the same cluster or observational unit to remain statistically dependent through shared latent variables.

GLMMs extend ordinary linear mixed models to outcomes for which conditional means are not adequately represented by an additive Gaussian model. Their defining distinction is that the linear predictor describes the response after conditioning on the random effects. Integration over those effects generally produces a marginal mean and covariance structure that no longer has the same algebraic form as the conditional model.

Mathematical formulation

Let (Y_{ij}) denote the (j)-th response associated with cluster (i), and let (\mathbf b_i) be a vector of random effects. Conditional on (\mathbf b_i), the response follows an exponential-family distribution with mean

[ \mu_{ij}(\mathbf b_i) = \operatorname{E}(Y_{ij}\mid \mathbf b_i). ]

A link function (g) relates this conditional mean to a linear predictor:

[ g!\left(\mu_{ij}(\mathbf b_i)\right) = \mathbf x_{ij}^{\mathsf T}\boldsymbol\beta

\mathbf z_{ij}^{\mathsf T}\mathbf b_i. ]

The vector (\boldsymbol\beta) contains fixed-effect coefficients shared across the population. The design vector (\mathbf x_{ij}) associates those coefficients with each observation. The random-effect vector (\mathbf b_i) describes cluster-specific deviations, while (\mathbf z_{ij}) determines which deviations enter a particular response.

A conventional specification assigns a multivariate normal distribution to the random effects:

[ \mathbf b_i \sim N(\mathbf 0,\mathbf D), ]

where (\mathbf D) is a covariance matrix governed by variance and correlation parameters. Normality is not logically required by the definition of a GLMM, but it yields a compact latent-variable representation and underlies much of the standard inferential theory.

Conditional independence is commonly expressed as

[ Y_{ij}\perp Y_{ik}\mid\mathbf b_i \qquad (j\ne k). ]

Thus, dependence among responses from the same cluster arises after the shared random effects are integrated out. The resulting marginal covariance reflects both the conditional response variance and heterogeneity represented by (\mathbf D).

Conditional and marginal interpretation

Fixed-effect coefficients in a GLMM ordinarily have a conditional, or cluster-specific, interpretation. In a model with a logit link, a coefficient describes the change in conditional log odds associated with a covariate while the random effect remains fixed. It does not generally equal the corresponding coefficient from a marginal model, because the logistic transformation is nonlinear and integration does not commute with that transformation.

This distinction can be written as

[ \operatorname{E}_{\mathbf b} \left[ g^{-1}!\left( \mathbf x^{\mathsf T}\boldsymbol\beta +\mathbf z^{\mathsf T}\mathbf b \right) \right] \ne g^{-1}!\left( \mathbf x^{\mathsf T}\boldsymbol\beta \right) ]

for most nonlinear inverse links. The left side is a population-averaged mean obtained by integrating over latent heterogeneity. The right side is the conditional mean evaluated at a zero random effect. Their difference is an intrinsic consequence of the model rather than an estimation artifact.

A random intercept allows clusters to differ in their baseline linear predictor. A random slope permits the association between a covariate and the response to vary across clusters. When both occur in the same model, their covariance represents the relation between baseline level and cluster-specific covariate response.

Likelihood and integration

For independent clusters, the marginal likelihood is

[ L(\boldsymbol\beta,\boldsymbol\theta)

\prod_i \int \left[ \prod_j p!\left( y_{ij}\mid\mathbf b_i,\boldsymbol\beta \right) \right] f!\left( \mathbf b_i\mid\boldsymbol\theta \right) ,d\mathbf b_i, ]

where (\boldsymbol\theta) parameterizes the random-effect distribution. The integral removes the unobserved (\mathbf b_i) and connects the conditional model to the observed-data likelihood.

Unlike the Gaussian linear mixed model, most GLMMs do not yield a closed-form marginal likelihood. The principal computational problem is therefore the evaluation of a potentially high-dimensional integral whose dimension increases with the number of random effects assigned to each independent integration unit.

A Laplace approximation expands the log-integrand around the conditional mode of the random effects. Its first-order form is computationally compact, although its error can become material when cluster information is limited or the conditional likelihood is strongly asymmetric. Adaptive Gaussian quadrature places numerical integration nodes relative to the conditional mode and curvature, providing a more detailed representation of low-dimensional integrals at increasing computational cost.

Breslow and Clayton developed the penalized quasi-likelihood formulation that expresses GLMM estimation through iteratively updated working mixed models. This construction connected generalized estimating equations with the computational machinery of linear mixed models. Its lower-order approximations can produce appreciable bias in variance components when responses are highly discrete and little information is available within clusters.

Monte Carlo integration and Markov chain Monte Carlo provide simulation-based alternatives. In a Bayesian formulation, prior distributions are assigned to fixed effects and variance parameters, and inference proceeds from the joint posterior distribution. The statistical model remains a GLMM when the same conditional response law, link function, linear predictor, and random-effect structure are retained.

Historical development

The GLMM arose from the convergence of two established modeling traditions. Charles Roy Henderson developed mixed-model equations for Gaussian data with random effects, providing an algebraic basis for prediction and variance-component estimation. John Nelder and Robert Wedderburn later formalized generalized linear models through exponential-family distributions, linear predictors, and link functions.

During the late 1980s, You Watanabe derived a conditional-mode block expansion for clustered logistic mixed models. The expansion separated contributions from independent clusters before applying a second-order approximation to each random-effect integral. This formulation reduced the original observed-data likelihood to a sequence of local quadratic forms and became part of the computational lineage leading to blockwise Laplace evaluation.

Subsequent work placed these developments within a unified likelihood framework. The resulting class included models for repeated categorical responses, clustered event counts, and other non-Gaussian measurements while preserving the distinction between population-level coefficients and latent unit-level variation.

Identifiability and covariance structure

The random-effect covariance matrix must be compatible with the information contained in the grouping structure. A model that assigns multiple correlated random coefficients to clusters with few informative observations can approach the boundary of the parameter space. At that boundary, one variance estimate becomes zero or a correlation approaches unit magnitude, producing what is commonly called a singular fit.

Such behavior does not constitute a separate distributional assumption. It indicates that the fitted covariance structure has lower effective rank than its nominal parameterization. The likelihood may then identify a reduced random-effect subspace while remaining nearly flat in other directions.

Confounding can also occur between random-effect variation and conditional overdispersion. In a count model, excess marginal variability can arise from latent cluster heterogeneity, from a conditional distribution whose variance exceeds its mean, or from both mechanisms simultaneously. Their separation depends on repeated information and on how each source of variation induces covariance across observations.

Prediction and residual structure

Prediction of a random effect is based on its conditional distribution given the observed responses. The resulting empirical Bayes predictor is pulled toward the population mean because it combines cluster-specific information with the estimated random-effect distribution. The extent of this shrinkage depends on the amount of information within the cluster and on the estimated variance components.

Residuals in a GLMM do not have a single universal definition. Conditional residuals compare responses with fitted means that include predicted random effects. Marginal residuals instead compare responses with means integrated over the random-effect distribution. Randomized quantile residuals map discrete conditional distributions onto a continuous reference scale by incorporating the probability mass assigned to each observed response.

Model discrepancy can appear through the mean structure, through the conditional response distribution, or through the random-effect specification. These components affect different parts of the implied joint distribution, although finite datasets can leave their empirical manifestations partially confounded.

Relation to neighboring model classes

A GLMM differs from a generalized estimating equation model in its primary representation of dependence. The GLMM introduces latent random effects and thereby specifies a full conditional probability model. A generalized estimating equation instead characterizes the marginal mean and uses a working correlation structure without requiring a complete joint distribution.

A hierarchical generalized linear model broadens the hierarchy by assigning distributions to additional model components and by treating multiple linked linear predictors within one framework. A generalized additive mixed model replaces selected linear covariate effects with smooth functions while retaining random effects and a non-Gaussian response model.

The GLMM can also be represented as a latent variable model. Under this interpretation, the random effects are unobserved variables that induce association among conditionally independent responses. This representation connects GLMMs to multilevel models, item-response models, and shared-frailty survival models, although their sampling distributions and inferential targets differ.