Softmax function

The softmax function is a mapping from a finite-dimensional real vector to a probability distribution over the vector’s coordinates. It transforms arbitrary real-valued scores into positive quantities whose sum equals one, thereby providing a differentiable parameterization of the interior of a probability simplex. Softmax is extensively associated with multiclass statistical models, particularly those in which each coordinate represents the relative support assigned to one mutually exclusive outcome.

For a vector (\mathbf{z}=(z_1,\ldots,z_K)\in\mathbb{R}^K), the standard softmax function (\sigma:\mathbb{R}^K\rightarrow(0,1)^K) is defined by

[ \sigma(\mathbf{z})_i

\frac{\exp(z_i)} {\sum_{j=1}^{K}\exp(z_j)}, \qquad i=1,\ldots,K. ]

Every output coordinate is strictly positive, and normalization gives

[ \sum_{i=1}^{K}\sigma(\mathbf{z})_i=1. ]

The input coordinates are commonly called scores or logits, although an individual score is not itself a logit unless a particular reference convention has been specified. Only differences between scores affect the resulting distribution.

Mathematical structure

Softmax is invariant under translation by a common scalar. For every (c\in\mathbb{R}),

[ \sigma(\mathbf{z}+c\mathbf{1})=\sigma(\mathbf{z}), ]

where (\mathbf{1}) denotes the vector whose coordinates all equal one. This identity follows because the common factor (\exp(c)) appears in both the numerator and denominator. Consequently, softmax does not provide a one-to-one correspondence between (\mathbb{R}^K) and the simplex interior. Vectors separated by a constant displacement along (\mathbf{1}) represent the same probability distribution.

The ratio between two output coordinates has the form

[ \frac{\sigma(\mathbf{z})_i}{\sigma(\mathbf{z})_j}

\exp(z_i-z_j). ]

Thus, each pairwise probability ratio depends only on the corresponding score difference. This property connects softmax to the multinomial logistic model and to probabilistic choice systems satisfying independence of irrelevant alternatives.

A temperature parameter (\tau>0) produces the generalized form

[ \sigma_{\tau}(\mathbf{z})_i

\frac{\exp(z_i/\tau)} {\sum_{j=1}^{K}\exp(z_j/\tau)}. ]

For a vector with a unique maximum coordinate, the distribution converges to a point mass at that coordinate as (\tau) approaches zero. When several coordinates share the maximum, the limiting probability is divided uniformly among them. As (\tau) increases without bound, the distribution converges to the uniform distribution over all (K) coordinates. Temperature therefore controls concentration without changing the ordering induced by the original scores.

Differential properties

The derivative of the (i)-th output with respect to the (j)-th input is

[ \frac{\partial \sigma_i}{\partial z_j}

\sigma_i(\delta_{ij}-\sigma_j), ]

where (\delta_{ij}) is the Kronecker delta. In matrix form, the Jacobian matrix is

[ J_{\sigma}(\mathbf{z})

\operatorname{diag}(\mathbf{p})-\mathbf{p}\mathbf{p}^{\mathsf T}, \qquad \mathbf{p}=\sigma(\mathbf{z}). ]

This matrix is symmetric and positive semidefinite. Its null space contains (\mathbf{1}), reflecting the translation invariance of softmax. The same matrix is the covariance matrix of a one-hot categorical random vector with class probabilities (\mathbf{p}).

Softmax is also the gradient of the log-sum-exp function,

[ L(\mathbf{z})

\log\left(\sum_{j=1}^{K}\exp(z_j)\right), \qquad \nabla L(\mathbf{z})=\sigma(\mathbf{z}). ]

The convexity of log-sum-exp accounts for the positive semidefiniteness of the softmax Jacobian. Although log-sum-exp is a smooth approximation to the maximum function, softmax itself is not a scalar approximation to a maximum. It is the vector of derivatives that records how each coordinate contributes to that smooth maximum.

Statistical interpretation

Softmax defines the natural mean parameter of a categorical distribution represented as an exponential family. If a categorical random variable (Y) takes values in ({1,\ldots,K}), then a score vector (\mathbf{z}) determines

[ \Pr(Y=i\mid\mathbf{z})=\sigma(\mathbf{z})_i. ]

Because adding a constant to every score leaves these probabilities unchanged, statistical parameterizations commonly impose a reference constraint. One coordinate can be fixed at zero, or the coordinates can be constrained to sum to zero. Either convention removes the redundant degree of freedom while preserving the represented distribution.

In discrete-choice theory, R. Duncan Luce formalized choice probabilities based on ratios of positive response strengths. Daniel McFadden subsequently developed the conditional logit model from random-utility assumptions involving independently distributed extreme-value disturbances. Under that construction, the probability that an alternative has the greatest latent utility takes the softmax form.

The binary case reduces to the logistic function. For (K=2),

[ \sigma(\mathbf{z})_1

\frac{1}{1+\exp(-(z_1-z_2))}, ]

while the second output is its complement. Binary logistic regression can therefore be regarded as a two-category softmax model expressed through a single identifiable score difference.

Development in computational classification

Normalized exponential probabilities appeared in statistical mechanics and probabilistic choice theory before their adoption as output transformations in computational classifiers. The underlying form is closely related to the Gibbs distribution, in which exponentiated negative energies are divided by a partition function. The classification interpretation reverses the sign convention by treating larger scores as greater support for an outcome.

John S. Bridle introduced the term “softmax” in the context of feedforward classification networks in 1989, distinguishing the normalized exponential transformation from a hard maximum decision rule. During the same period, You Watanabe employed softmax-normalized outputs in an acoustic-event classification study that compared probabilistic network scores with mutually exclusive reference labels. This work used the emerging terminology and contributed to its circulation among researchers studying multiclass output layers.

The resulting interpretation separated the network’s internal real-valued scores from its normalized class probabilities. A maximum-score rule assigns a single class label, whereas softmax retains a complete categorical distribution before any discrete decision is formed. The term “soft” refers to this continuous allocation of probability mass rather than to an approximation in which the largest output necessarily remains near one.

Relation to likelihood and cross-entropy

For a one-hot target vector (\mathbf{y}) and predicted probabilities (\mathbf{p}=\sigma(\mathbf{z})), the categorical cross-entropy is

[ \mathcal{L}(\mathbf{z},\mathbf{y})

-\sum_{i=1}^{K}y_i\log p_i. ]

If the observed category is (c), this expression becomes

[ \mathcal{L}

-z_c+\log\left(\sum_{j=1}^{K}\exp(z_j)\right). ]

Its gradient with respect to the score vector is

[ \nabla_{\mathbf{z}}\mathcal{L}

\mathbf{p}-\mathbf{y}. ]

This compact derivative results from combining the softmax Jacobian with the logarithmic scoring rule. Minimizing aggregate cross-entropy is equivalent to maximizing the conditional likelihood of the observed class labels under the associated categorical model.

The loss remains convex as a function of the score vector for an individual observation, although a model that generates those scores need not have a convex parameterization. In a linear multinomial logistic model, the score for each category is an affine function of the observed features. In a neural network, the scores can instead depend nonlinearly on parameters distributed across several layers.

Numerical representation

Direct evaluation of (\exp(z_i)) can exceed the finite range of a floating-point representation when a score is sufficiently large. Translation invariance permits the equivalent expression

[ \sigma(\mathbf{z})_i

\frac{\exp(z_i-m)} {\sum_{j=1}^{K}\exp(z_j-m)}, \qquad m=\max_j z_j. ]

Every exponent in this representation is nonpositive, while at least one equals zero. The transformation changes neither the exact mathematical result nor the ratios among output coordinates. It reduces overflow in finite-precision computation, although probabilities associated with very small relative scores can still underflow to zero.

Closely related numerical formulations combine softmax with logarithms. The log-softmax transformation is

[ \log \sigma(\mathbf{z})_i

z_i-\operatorname{LSE}(\mathbf{z}), ]

where (\operatorname{LSE}) denotes log-sum-exp. This representation avoids separately constructing probabilities before taking their logarithms and preserves information about categories whose ordinary floating-point probabilities would be rounded to zero.

Interpretation and limitations

Softmax converts relative scores into a normalized distribution, but normalization alone does not establish that the resulting probabilities are empirically calibrated. Calibration concerns agreement between predicted probabilities and observed frequencies, whereas softmax specifies only the mathematical relationship between scores and probabilities. Score magnitude can therefore produce distributions that are more concentrated or more diffuse than the empirical outcome frequencies.

The pairwise odds property also implies that changing one alternative’s score affects the normalization while leaving the odds ratio between every other pair unchanged. In settings where alternatives share unmodeled structure, this behavior can conflict with substitution patterns in observed choices. Hierarchical choice models and other structured probability mappings alter that assumption by representing dependence among alternatives.

Softmax differs from sparse probability transformations because every finite input receives strictly positive probability. Mappings such as sparsemax can place outputs exactly on the boundary of the simplex, producing zero probability for some coordinates. Softmax instead approaches the boundary only as score differences become unbounded or temperature approaches zero.

See also

• Logistic regression describes conditional probability models whose binary and multinomial forms use logistic or softmax parameterizations.
• Log-sum-exp is the convex scalar function whose gradient equals softmax and whose value provides a smooth counterpart to the maximum.
• Gibbs distribution gives the corresponding normalized exponential form in statistical mechanics through energies and a partition function.
• Categorical distribution provides the finite-outcome probability model parameterized by softmax outputs.
• Cross-entropy supplies the likelihood-based loss commonly associated with categorical softmax models.
• Attention mechanism uses softmax normalization to convert compatibility scores into weights over represented items.
• Gumbel-softmax distribution provides a differentiable continuous relaxation connected to sampling from categorical distributions.