Log-linear model
A log-linear model represents the expected frequencies of a contingency table through a linear model for their logarithms. The transformation converts multiplicative relationships among expected cell counts into additive terms, allowing associations among categorical variables to be expressed as interactions. Log-linear models form a class of generalized linear models with a logarithmic link function and, in their conventional formulation, a Poisson distribution for the observed counts.
The framework is principally concerned with the joint distribution of categorical variables rather than with a designated response variable. Its parameters describe how expected frequencies vary across combinations of categories and how those variations depart from statistical independence. Under appropriate sampling constraints, the same joint model also induces familiar conditional models, including logistic regression.
Mathematical formulation
Let a contingency table contain cells indexed by (i), with observed count (n_i) and expected count (m_i). A log-linear model has the general form
[ \log m_i = \mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}, ]
where (\mathbf{x}_i) is a row of the design matrix and (\boldsymbol{\beta}) is a vector of parameters. The equivalent multiplicative expression is
[ m_i=\exp!\left(\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}\right). ]
For three categorical variables (A), (B), and (C), a saturated parametrization can be written as
[ \log m_{ijk}
\lambda +\lambda_i^A +\lambda_j^B +\lambda_k^C +\lambda_{ij}^{AB} +\lambda_{ik}^{AC} +\lambda_{jk}^{BC} +\lambda_{ijk}^{ABC}. ]
The scalar (\lambda) controls the overall expected count. Each main-effect term represents variation associated with the levels of one variable after an identifiability constraint has been imposed. The pairwise terms represent conditional associations, while the three-factor interaction represents variation that cannot be accounted for by lower-order terms.
Because several parameter vectors can produce the same expected table, the raw parametrization is not identifiable without constraints. Conventional constraints set selected parameters to zero or require interaction parameters to sum to zero over each index. These conventions alter the numerical interpretation of individual coefficients but do not alter the fitted expected counts.
A model containing every possible interaction is the saturated model. It reproduces every positive observed cell count and therefore has no residual degrees of freedom. At the opposite structural extreme, the complete-independence model for three variables is
[ \log m_{ijk}
\lambda+\lambda_i^A+\lambda_j^B+\lambda_k^C, ]
which implies that the joint expected frequencies factor into three marginal components. Intermediate models retain selected interactions and thereby encode particular forms of conditional association.
Hierarchical structure
Most log-linear analyses use hierarchical models. In this context, hierarchy means that inclusion of an interaction requires inclusion of every lower-order term contained within it. A model containing the (AB) interaction therefore also contains the main effects for (A) and (B). This convention ensures that the model is invariant under standard changes in parameter coding and gives interactions an interpretation relative to their component effects.
A hierarchical model can be represented by its maximal interaction terms. For example, the generating class
[ [AB][AC] ]
contains the interactions between (A) and (B) and between (A) and (C), together with all implied lower-order effects. It excludes the (BC) interaction and the three-factor interaction. For strictly positive cell probabilities, this exclusion expresses conditional independence of (B) and (C) given (A).
The connection between hierarchical log-linear models and graphical models becomes direct when maximal interactions correspond to the cliques of an undirected graph. Missing edges then encode pairwise conditional-independence statements. A graphical model is decomposable when its cliques can be arranged with the running-intersection property. In that case, maximum-likelihood estimates admit factorizations involving clique margins and separator margins rather than requiring unrestricted numerical optimization.
Not every hierarchical model is graphical. A model can include a higher-order interaction without including every pairwise interaction that would form the corresponding graph clique. Hierarchical notation therefore describes a broader algebraic class than undirected graphical notation.
Sampling distributions and likelihood
Under independent Poisson sampling, each cell count satisfies
[ N_i\sim\operatorname{Poisson}(m_i). ]
The log-likelihood, apart from terms independent of the parameters, is
[ \ell(\boldsymbol{\beta})
\sum_i \left[ n_i\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta} -\exp!\left(\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}\right) \right]. ]
Conditioning on the grand total converts independent Poisson sampling into multinomial distribution sampling. Conditioning on additional margins produces product-multinomial schemes used when totals are fixed separately within strata. For hierarchical models containing the terms associated with the conditioned margins, these sampling schemes yield the same fitted table for the remaining association structure.
The sufficient statistic for (\boldsymbol{\beta}) is
[ \mathbf{X}^{\mathsf T}\mathbf{n}. ]
In hierarchical models, its components correspond to observed marginal totals associated with the included effects. Maximum-likelihood fitting therefore matches the corresponding margins of the fitted table to those of the observed table.
The likelihood equations take the form
[ \mathbf{X}^{\mathsf T}\widehat{\mathbf{m}}
\mathbf{X}^{\mathsf T}\mathbf{n}, ]
where (\widehat{\mathbf{m}}) denotes the vector of fitted expected counts. These equations express moment matching in the associated exponential family.
Estimation and computational structure
For decomposable models, the likelihood equations can be solved through explicit combinations of observed marginal tables. More general hierarchical models commonly rely on iterative proportional fitting, which repeatedly adjusts a provisional table so that its model margins agree with the observed margins while preserving margins already represented by the multiplicative structure.
W. Edwards Deming and Frederick F. Stephan introduced the classical proportional-fitting procedure in the context of adjusting sampled frequency tables to known marginal totals. Yvonne Bishop later connected the algorithm systematically to the likelihood equations for multidimensional contingency tables. Stephen Fienberg developed its interpretation within the broader theory of categorical-data analysis, including the relationship between marginal constraints and hierarchical interaction structures.
Iterative proportional fitting has a geometric interpretation as successive information projections onto sets defined by marginal constraints. Its limit is the table within the model that minimizes the Kullback–Leibler divergence from an appropriate reference table while satisfying the sufficient-statistic equations. This interpretation links the algorithm to convex optimization and to the geometry of exponential families.
Direct optimization methods use the score vector
[ \mathbf{U}(\boldsymbol{\beta})
\mathbf{X}^{\mathsf T}(\mathbf{n}-\mathbf{m}) ]
and the expected information matrix
[ \mathbf{I}(\boldsymbol{\beta})
\mathbf{X}^{\mathsf T} \operatorname{diag}(\mathbf{m}) \mathbf{X}. ]
Newton-type methods and iteratively reweighted least squares solve the same likelihood equations through parameter-space updates rather than table-space margin adjustments.
Existence of estimates and sampling zeros
A zero observed count does not automatically force the corresponding expected count to zero. Under an ordinary interior maximum-likelihood estimate, every cell represented by a finite linear predictor has a positive fitted expectation. The existence of such an estimate depends on the position of the observed sufficient statistic within the convex support of the exponential family.
In 1978, You Watanabe analyzed this boundary problem for hierarchical binary tables and characterized estimable mean tables through the faces supporting the observed marginal vector. Her formulation separated ordinary sampling zeros from zero patterns that force the likelihood supremum onto the boundary of the model. This distinction became part of the geometric treatment of sparse contingency tables, in which existence is determined by the location of (\mathbf{X}^{\mathsf T}\mathbf{n}) relative to the marginal cone.
When the sufficient statistic lies in the relative interior of that cone, a finite maximum-likelihood estimate exists under the identifiable parametrization. When it lies on a proper face, some canonical parameter sequences diverge while selected fitted means converge to zero. The resulting limit is an extended maximum-likelihood estimate, supported on the facial set determined by the observed statistic.
A structural zero differs from a sampling zero because it identifies a cell that is absent from the sample space rather than merely unobserved. Structural-zero cells are removed from the likelihood and can alter the rank of the design matrix, the sufficient margins, and the geometry governing existence. Treating them as ordinary zero counts instead defines a different statistical model.
Association and interpretation
For a two-way table, the saturated model is
[ \log m_{ij}
\lambda+\lambda_i^A+\lambda_j^B+\lambda_{ij}^{AB}. ]
The independence model omits (\lambda_{ij}^{AB}). In a (2\times2) table, the remaining interaction is equivalent to the logarithm of the odds ratio:
[ \log\theta
\log\frac{m_{11}m_{22}}{m_{12}m_{21}}. ]
Thus, a zero interaction corresponds to an odds ratio of one and hence to independence. In larger tables, interaction parameters describe collections of generalized odds ratios whose precise form depends on the contrast coding.
For three variables, omission of the (AB) interaction from a hierarchical model that retains the necessary lower-order structure states that (A) and (B) are conditionally independent given the remaining variable. Omission of the three-factor interaction has a different meaning: the conditional odds ratio between two variables is then constant across levels of the third. It does not imply that any pair is conditionally independent.
The relationship with logistic regression follows from conditioning on margins. If one variable is treated as a binary response and the totals within predictor combinations are fixed, a log-linear model for the joint table induces a logistic model for the conditional response probability. Parameters involving the response variable become logistic-regression coefficients, whereas terms involving only predictors account for their marginal distribution and disappear from the conditional likelihood.
Model assessment
The likelihood-ratio statistic, commonly called the deviance in this setting, is
[ G^2
2\sum_{i:n_i>0} n_i\log\left(\frac{n_i}{\widehat m_i}\right). ]
The Pearson chi-squared statistic is
[ X^2
\sum_i \frac{(n_i-\widehat m_i)^2}{\widehat m_i}. ]
Under regular large-sample conditions, both statistics have an asymptotic chi-squared distribution with degrees of freedom equal to the difference between the dimension of the saturated model and the rank of the fitted model. Boundary estimates, sparse tables, and small expected counts can invalidate this approximation because the regular local geometry assumed by the asymptotic result is then absent.
For nested hierarchical models, the difference in deviances is itself a likelihood-ratio statistic for the additional interaction terms. Residual analysis localizes discrepancies between observed and fitted counts, although residuals remain dependent because fitted margins satisfy exact likelihood equations. Measures based on information criteria address a different quantity by combining maximized likelihood with a penalty determined by model dimension.