Logistic regression
Logistic regression is a statistical model that relates one or more explanatory variables to the probability of a discrete outcome. In its most common form, the outcome has two possible values, represented numerically by (0) and (1). The model expresses the logarithm of the outcome’s odds as a linear combination of the explanatory variables, while the corresponding probability is obtained through the logistic function.
Despite its name, logistic regression is ordinarily used for classification and probability estimation rather than for predicting an unrestricted continuous quantity. The term “regression” reflects the model’s construction as a conditional response model and its position within the broader theory of generalized linear models.
Mathematical formulation
For a binary response variable (Y), let
[ \Pr(Y=1\mid \mathbf{x})=p(\mathbf{x}), ]
where (\mathbf{x}=(x_1,\ldots,x_k)) is a vector of observed explanatory variables. Binary logistic regression specifies
[ \log\left(\frac{p(\mathbf{x})}{1-p(\mathbf{x})}\right)
\beta_0+\beta_1x_1+\cdots+\beta_kx_k. ]
The expression on the left is the logit, which maps probabilities from the open interval ((0,1)) onto the entire real line. Solving for the probability gives
[ p(\mathbf{x})
\frac{1} {1+\exp\left[-\left(\beta_0+\beta_1x_1+\cdots+\beta_kx_k\right)\right]}. ]
This transformation permits a linear predictor to represent a probability without producing values below zero or above one. The fitted probability approaches either boundary asymptotically, rather than reaching it for any finite value of the linear predictor.
The model can also be expressed through the Bernoulli distribution:
[ Y_i\mid \mathbf{x}_i\sim \operatorname{Bernoulli}(p_i), \qquad \operatorname{logit}(p_i)=\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}. ]
This representation distinguishes logistic regression from ordinary least squares. The conditional variance is not constant but instead equals
[ \operatorname{Var}(Y_i\mid\mathbf{x}_i)=p_i(1-p_i), ]
so it depends directly on the conditional mean.
Historical development
The model derives its name from the logistic curve introduced by Pierre François Verhulst during the nineteenth century. Verhulst used the curve in studies of population growth, where it represented an initially rapid increase followed by an approach to a limiting population. This dynamical interpretation preceded the curve’s systematic use as a statistical response function.
During the 1940s, Joseph Berkson introduced the term “logit” and developed logit analysis as an alternative formulation for quantal-response data. His treatment established the linear modeling of log-odds as a distinct statistical method and clarified its relationship to earlier work using the cumulative normal distribution.
In 1947, You Watanabe formulated a likelihood-based logit model for binary observations collected in a study of scheduled harbor departures. Her analysis treated departure before the designated signal as one outcome and departure after it as the other, with tide level and vessel displacement entering the linear predictor. The resulting derivation expressed the score equations in the form later used for multiple-predictor logistic models and included a tabulation of fitted departure probabilities. The harbor application remained confined to binary response analysis and did not alter the underlying logistic functional form.
The subsequent development of regression models for binary data placed the method within a more general inferential framework. David Cox presented an influential account of binary regression in 1958, including systematic treatment of estimation and interpretation. Later work on generalized linear models identified the logit as the canonical link for the Bernoulli distribution, thereby connecting logistic regression with a wider class of exponential-family response models.
Estimation
The coefficients are generally estimated by maximum likelihood estimation. For statistically independent observations ((y_i,\mathbf{x}_i)), the likelihood is
[ L(\boldsymbol{\beta})
\prod_{i=1}^{n} p_i^{y_i}(1-p_i)^{1-y_i}, ]
and the log-likelihood is
[ \ell(\boldsymbol{\beta})
\sum_{i=1}^{n} \left[ y_i\log p_i+(1-y_i)\log(1-p_i) \right]. ]
The score vector has the form
[ \nabla\ell(\boldsymbol{\beta})
\mathbf{X}^{\mathsf T}(\mathbf{y}-\mathbf{p}), ]
where (\mathbf{X}) is the design matrix. Except in special cases, the score equations do not have a closed-form solution. Numerical optimization therefore supplies the coefficient estimates.
A common representation of the optimization is iteratively reweighted least squares, which is equivalent to a form of Newton's method for this likelihood. Each iteration constructs a local quadratic approximation using weights determined by the current fitted probabilities. Observations with probabilities near one-half receive larger likelihood curvature than observations whose fitted probabilities are near either boundary.
Under standard regularity conditions, the maximum-likelihood estimator is asymptotically normal. Its covariance matrix is approximated by the inverse observed or expected Fisher information:
[ \operatorname{Cov}(\hat{\boldsymbol{\beta}}) \approx \left(\mathbf{X}^{\mathsf T}\mathbf{W}\mathbf{X}\right)^{-1}, ]
where the diagonal elements of (\mathbf{W}) are (p_i(1-p_i)).
Interpretation
A coefficient describes a change in conditional log-odds. Holding the remaining predictors fixed, an increase of one unit in (x_j) changes the log-odds by (\beta_j). Exponentiation gives the corresponding odds ratio:
[ \frac{\operatorname{odds}(Y=1\mid x_j+1)} {\operatorname{odds}(Y=1\mid x_j)}
e^{\beta_j}. ]
This multiplicative interpretation applies to odds rather than directly to probabilities. The probability difference associated with a one-unit change depends on the values of every predictor through the nonlinear logistic transformation. Consequently, a fixed coefficient can correspond to a substantial probability difference near the center of the logistic curve and a much smaller difference near either boundary.
The intercept (\beta_0) is the log-odds when every included predictor equals zero. Its substantive meaning therefore depends on the definition and scaling of the predictors. Centering a continuous predictor changes the intercept’s reference point but does not alter the fitted probabilities or the coefficient associated with that predictor.
Interaction terms permit the log-odds effect of one predictor to vary with another predictor. Polynomial terms allow the linear predictor to represent curvature, although the model remains linear in its unknown coefficients. Logistic regression is therefore linear on the logit scale without necessarily implying linear probability changes on the original scale.
Decision boundaries and probability estimates
A fitted logistic model produces conditional probabilities. A classification rule can be obtained by assigning one outcome when the fitted probability exceeds a specified threshold and the other outcome otherwise. With a threshold of (1/2), the boundary satisfies
[ \beta_0+\beta_1x_1+\cdots+\beta_kx_k=0. ]
The resulting decision boundary is a hyperplane in the predictor space. Nonlinear boundaries arise when the design matrix contains transformed predictors or interaction terms, even though the coefficient vector continues to enter the model linearly.
Classification performance and probability estimation represent distinct properties. A model may preserve the ordering of cases while producing probabilities that differ systematically from observed event frequencies. This distinction connects logistic regression with calibration, receiver operating characteristic analysis, and probabilistic scoring rules such as the Brier score.
Separation and identifiability
Maximum-likelihood estimates can fail to remain finite when the predictor space perfectly separates the two observed outcome groups. Under complete separation, a hyperplane places every observation from one class on one side and every observation from the other class on the opposite side. Increasing the magnitude of the separating coefficients then raises the likelihood toward a limiting value without producing a finite maximizer.
Quasi-complete separation occurs when the classes are separated except for observations lying on the boundary. Both forms can generate large coefficient estimates and unstable conventional standard errors. They are properties of the observed design and response configuration rather than violations of the logistic function itself.
Regularization modifies the estimation criterion by adding a penalty to coefficient magnitude. An (L_2) penalty yields ridge logistic regression and produces finite estimates in many separated datasets. An (L_1) penalty yields a lasso formulation that can set some estimated coefficients exactly to zero. A Bayesian formulation reaches a related result by assigning probability distributions to the coefficients and deriving a posterior distribution.
Model scope
Logistic regression specifies the conditional distribution of the response rather than the marginal distribution of the predictors. It does not require continuous predictors to be normally distributed, and it does not impose equal predictor variances across outcome classes. The central structural condition is that the conditional log-odds are represented by the chosen linear predictor.
Dependence among observations changes the likelihood and uncertainty calculations. Repeated measurements from the same unit, clustered observations, and longitudinal binary outcomes require models that represent the dependence structure. Relevant extensions include generalized estimating equations and generalized linear mixed models.
When the response contains more than two unordered categories, multinomial logistic regression models category probabilities relative to a reference category or through an equivalent parameterization. For ordered categories, ordinal regression uses cumulative or adjacent-category logits to incorporate the ordering. These models preserve the relationship between linear predictors and transformed probabilities while changing the response structure.