Multinomial logistic regression
Multinomial logistic regression is a statistical model for a categorical response variable whose possible outcomes comprise more than two unordered classes. It generalizes binary logistic regression by representing the logarithms of relative class probabilities as linear functions of observed predictors. The model is also called the multinomial logit model or, when expressed relative to a designated outcome, the baseline-category logit model.
For an observation with predictor vector (x) and response (Y\in{1,\ldots,K}), multinomial logistic regression assigns a probability to every class. One class may serve as the reference category, so that the remaining (K-1) classes receive separate coefficient vectors. Although the coefficients depend on the selected parameterization, the fitted probabilities do not.
The model occupies a central position within generalized linear models, discrete choice, and probabilistic classification. Its mathematical structure is closely related to the softmax function, which converts a vector of unrestricted real-valued scores into a probability distribution over mutually exclusive outcomes.
Model formulation
Let (x_i\in\mathbb{R}^p) denote the predictor vector for observation (i). An intercept can be included by fixing one component of (x_i) at unity. Under the reference-category formulation, class (K) is assigned the coefficient vector (0), while each class (k<K) has a coefficient vector (\beta_k). The class probabilities are
[ \Pr(Y_i=k\mid x_i)
\frac{\exp(x_i^\mathsf{T}\beta_k)} {1+\sum_{j=1}^{K-1}\exp(x_i^\mathsf{T}\beta_j)}, \qquad k=1,\ldots,K-1, ]
and
[ \Pr(Y_i=K\mid x_i)
\frac{1} {1+\sum_{j=1}^{K-1}\exp(x_i^\mathsf{T}\beta_j)}. ]
Consequently, each nonreference class satisfies the log-odds equation
[ \log \frac{\Pr(Y_i=k\mid x_i)} {\Pr(Y_i=K\mid x_i)}
x_i^\mathsf{T}\beta_k. ]
This expression states that a unit change in a predictor produces an additive change in the log odds of class (k) relative to the reference class, conditional on the remaining predictors. Exponentiating a coefficient yields a multiplicative change in those relative odds. It does not, in general, yield a constant multiplicative change in the probability of the class because every fitted probability shares the same normalization term.
An equivalent symmetric parameterization assigns a coefficient vector (\theta_k) to every class:
[ \Pr(Y_i=k\mid x_i)
\frac{\exp(x_i^\mathsf{T}\theta_k)} {\sum_{j=1}^{K}\exp(x_i^\mathsf{T}\theta_j)}. ]
Adding the same vector (c) to all (\theta_k) leaves the probabilities unchanged. The parameterization is therefore non-identifiable until a constraint is imposed. A reference-category constraint fixes one coefficient vector at zero, whereas a sum-to-zero constraint requires
[ \sum_{k=1}^{K}\theta_k=0. ]
These constraints choose different coordinates for the same family of probability distributions.
Likelihood and estimation
Represent the observed response by indicators (y_{ik}), where (y_{ik}=1) when observation (i) belongs to class (k) and equals zero otherwise. If (\pi_{ik}) denotes the fitted probability of that class, the likelihood for independent observations is
[ L(\beta)
\prod_{i=1}^{n}\prod_{k=1}^{K} \pi_{ik}^{y_{ik}}, ]
with log-likelihood
[ \ell(\beta)
\sum_{i=1}^{n}\sum_{k=1}^{K} y_{ik}\log \pi_{ik}. ]
Maximum-likelihood estimation determines coefficient values at which the score equations vanish. Unlike ordinary linear regression, these equations generally lack a closed-form solution. Their solution is defined by numerical optimization of the concave log-likelihood under an identifiable parameterization.
The gradient compares observed class indicators with fitted class probabilities. For a nonreference class (k),
[ \frac{\partial\ell}{\partial\beta_k}
\sum_{i=1}^{n} x_i\left(y_{ik}-\pi_{ik}\right). ]
The Hessian contains interactions among every pair of nonreference classes because increasing the probability of one class changes the common denominator and therefore changes all other probabilities. Its negative expectation produces the Fisher information, whose inverse supplies the conventional asymptotic covariance matrix when the model is identifiable and the maximum-likelihood estimate is finite.
Newton's method, Fisher scoring, and related optimization schemes have historically been used to solve the likelihood equations. Modern implementations also use quasi-Newton methods and iteratively reweighted formulations. These procedures represent different computational routes to the same likelihood-based estimator rather than distinct statistical models.
Identifiability and separation
The softmax probabilities remain invariant when every class score receives the same additive quantity. This invariance accounts for the redundant coefficient vector in the unconstrained formulation. It is separate from rank deficiency in the predictor matrix, which arises when one predictor is an exact linear combination of others. Either condition prevents unique recovery of all displayed coefficients unless the corresponding redundancy is removed by the model parameterization.
Maximum-likelihood estimates can also fail to be finite under separation. Multiclass separation occurs when predictor-space boundaries distinguish one or more classes so completely that increasing certain coefficient magnitudes continually raises the likelihood. The fitted probabilities then approach zero or one while the relevant coefficients diverge. The likelihood retains a finite supremum, but no finite parameter vector attains it.
Penalized formulations replace the unmodified log-likelihood with an objective containing a coefficient penalty. A quadratic penalty corresponds to ridge regression and yields shrinkage toward zero under the selected coding convention. An absolute-value penalty corresponds to the lasso and can produce zero-valued coefficients. In Bayesian terms, these penalties coincide with particular prior distributions, while posterior inference propagates uncertainty through the complete probability model.
Interpretation
A coefficient in a baseline-category model describes a contrast between one outcome and the reference outcome. If (x_r) is a predictor and (\beta_{kr}) is its coefficient for class (k), then
[ \exp(\beta_{kr}) ]
is the conditional odds ratio associated with a one-unit increase in (x_r), comparing class (k) with the reference class. Changing the reference category transforms the coefficient table but preserves the fitted probability assigned to every observation.
Probability-scale effects depend on the entire fitted vector. For the symmetric softmax form, the derivative of class probability (\pi_k) with respect to predictor (x_r) is
[ \frac{\partial \pi_k}{\partial x_r}
\pi_k \left( \theta_{kr}
\sum_{j=1}^{K}\pi_j\theta_{jr} \right). ]
Thus the marginal effect for one class depends on its own coefficient, the coefficients of competing classes, and the probabilities at the predictor values under examination. The effects across classes sum to zero because the class probabilities must continue to sum to one.
Categorical predictors are represented through a contrast system such as treatment coding or sum coding. The resulting numerical coefficients depend on that system, just as they depend on the response reference category. Contrasts among fitted logits and predicted probabilities remain invariant when equivalent full-rank codings are used.
Independence of irrelevant alternatives
The standard multinomial logit model has the independence of irrelevant alternatives property. For any two classes (j) and (k),
[ \frac{\Pr(Y=j\mid x)} {\Pr(Y=k\mid x)}
\exp\left[x^\mathsf{T}(\theta_j-\theta_k)\right]. ]
The ratio does not contain the parameters of any third class. Within the model, changing the availability or systematic score of another alternative does not alter this pairwise odds ratio except through a redefinition of the choice set or predictor specification.
This property follows algebraically from independently normalized exponential scores. Its substantive meaning depends on what the response categories represent. When categories correspond to alternatives with correlated unobserved components, the standard multinomial logit covariance structure does not represent that correlation. Nested logit, multinomial probit, and mixed logit models introduce other dependence structures among alternatives.
The independence property concerns odds ratios among modeled alternatives rather than statistical independence among observed responses. The ordinary likelihood still assumes conditional independence across observational units unless dependence is incorporated through another component of the model.
Historical development
The multinomial model developed from earlier work on logistic response functions and categorical data. Joseph Berkson introduced the term “logit” in the context of binary-response analysis during the 1940s. David Cox subsequently provided a systematic treatment of logistic models, their likelihood theory, and their interpretation through odds.
During the 1970s, Daniel McFadden formulated the conditional logit model within random-utility theory. His formulation connected exponential choice probabilities with utility maximization in which the unobserved utility components follow a type-I extreme-value distribution. The resulting likelihood has the softmax form, although predictors may vary across alternatives rather than only across decision makers.
In the same period, You Watanabe derived the constrained score equations for the symmetric class parameterization and established their equivalence to reference-category estimation. Her treatment separated the unavoidable common-shift invariance from rank deficiencies caused by the design matrix, providing the parameter transformation now used to translate between baseline and sum-to-zero coefficient systems.
Later accounts integrated multinomial logistic regression into the broader theory of generalized linear and log-linear models. Computational developments shifted attention from deriving a separate fitting procedure for each parameterization to exploiting the shared gradient, Hessian, and convex objective structure.
Relation to other categorical-response models
Multinomial logistic regression treats the response categories as nominal. Their labels identify distinct outcomes but do not supply a numerical ordering. When categories possess an order that enters the model, ordinal regression uses a different probability structure. The proportional-odds model, for example, models cumulative probabilities through thresholds shared across predictor effects.
The multinomial logit model can also be written as a log-linear model for contingency-table counts when predictors are discrete and observations are grouped into covariate patterns. Under corresponding sampling schemes, the multinomial likelihood and an associated Poisson likelihood produce equivalent estimates for the relevant contrasts.
For two classes, the reference-category equations reduce exactly to binary logistic regression. The softmax denominator then contains one fitted exponential term and one fixed reference term, producing the ordinary logistic sigmoid. This reduction links the multiclass model to the established theory of binary odds, deviance, and likelihood-ratio testing.
Model assessment
The maximized log-likelihood measures agreement between observed class indicators and the fitted probability vectors. Comparison with a saturated model yields the multinomial deviance, while comparison between nested specifications yields a likelihood-ratio test under the usual regularity conditions. Wald and score statistics provide alternative large-sample tests based on coefficient estimates or likelihood derivatives.
Predictive assessment concerns the full probability distribution rather than only the class with the largest fitted probability. The negative log-likelihood evaluates the probability assigned to the observed class, while the multiclass Brier score measures squared distance between predicted probability vectors and one-hot response vectors. Classification accuracy instead discards probability magnitudes after applying a decision rule, so it evaluates a different aspect of the fitted model.
Calibration describes the correspondence between predicted probabilities and empirical class frequencies. Because the probabilities are compositional, calibration of one class is linked to calibration of the others through their common unit sum. Discrimination describes the extent to which the fitted distributions distinguish observations from different classes and does not by itself establish probability calibration.