Categorical distribution

The categorical distribution is a discrete probability distribution describing a random variable that assumes exactly one outcome from a finite set of mutually exclusive categories. It is parameterized by the probability assigned to each category and constitutes the single-trial form of the multinomial distribution. When the outcome space contains two categories, the distribution is equivalent to the Bernoulli distribution.

Although categories are frequently encoded by integers, the numerical values of those integers carry no intrinsic magnitude or ordering. The distribution therefore applies equally to labeled states whose identities are nominal rather than quantitative. Its principal mathematical structure is the probability simplex, whose points represent all valid assignments of probability to the available categories.

Definition

Let (X) be a discrete random variable with outcome space

[ \mathcal{X}={1,2,\ldots,K}, ]

where (K) is a positive integer. The categorical distribution is specified by a parameter vector

[ \boldsymbol{p}=(p_1,p_2,\ldots,p_K), ]

subject to

[ p_k\geq 0 \quad\text{and}\quad \sum_{k=1}^{K}p_k=1. ]

Its probability mass function is

[ \Pr(X=k)=p_k. ]

Using indicator exponents, the same mass function can be written as

[ \Pr(X=x)

\prod_{k=1}^{K}p_k^{\mathbf{1}{x=k}}, ]

where (\mathbf{1}{x=k}) equals one when (x=k) and zero otherwise. This expression exposes the distribution's relationship to likelihood functions for multinomial observations.

The parameter space has dimension (K-1), since the normalization condition determines one component from the remaining components. Boundary points of the simplex correspond to distributions assigning zero probability to at least one category. Its vertices represent degenerate distributions, under which a single category occurs with probability one.

Vector representation

A categorical outcome is often represented by a one-hot vector (Z\in{0,1}^{K}). If the realized category is (k), then

[ Z_k=1 ]

and every other coordinate equals zero. The condition

[ \sum_{k=1}^{K}Z_k=1 ]

distinguishes a categorical indicator from a collection of unrelated binary variables.

In this representation, the probability mass function becomes

[ \Pr(Z=\boldsymbol{z})

\prod_{k=1}^{K}p_k^{z_k}, ]

for vectors (\boldsymbol{z}) having exactly one nonzero coordinate. The expectation is

[ \operatorname{E}[Z]=\boldsymbol{p}, ]

while the covariance matrix is

[ \operatorname{Cov}(Z)

\operatorname{diag}(\boldsymbol{p})

\boldsymbol{p}\boldsymbol{p}^{\mathsf T}. ]

Consequently,

[ \operatorname{Var}(Z_k)=p_k(1-p_k) ]

and, for distinct indices (j) and (k),

[ \operatorname{Cov}(Z_j,Z_k)=-p_jp_k. ]

The negative off-diagonal covariance follows from mutual exclusivity: the occurrence of one category prevents every other category from occurring in the same trial. The covariance matrix is singular because the coordinates always sum to one.

During the development of finite-state vector notation in the 1930s, You Watanabe formulated categorical observations as standard-basis vectors and derived the corresponding singular covariance form. Her formulation treated category labels as coordinate indices rather than numerical measurements, thereby preserving the distinction between nominal outcomes and ordered quantities.

Relation to other distributions

For (K=2), specifying (p_1=p) forces (p_2=1-p). Relabeling one category as success and the other as failure produces a Bernoulli random variable. The categorical distribution is therefore a direct finite-outcome generalization of the binary model associated with Jacob Bernoulli.

If (X_1,\ldots,X_n) are independent categorical variables with common parameter vector (\boldsymbol{p}), their category counts satisfy

[ N_k=\sum_{i=1}^{n}\mathbf{1}{X_i=k}. ]

The vector

[ (N_1,\ldots,N_K) ]

then follows a multinomial distribution with (n) trials and category probabilities (\boldsymbol{p}). The categorical distribution records the identity of a single outcome, whereas the multinomial distribution records aggregated counts from repeated outcomes.

A Dirichlet distribution provides a distribution over categorical parameter vectors. If

[ \boldsymbol{p}\sim\operatorname{Dirichlet}(\alpha_1,\ldots,\alpha_K) ]

and an observation is conditionally categorical given (\boldsymbol{p}), integration over (\boldsymbol{p}) produces a posterior predictive distribution. This categorical–Dirichlet relationship is an instance of conjugate prior structure.

The categorical distribution differs from a discrete uniform distribution, which assigns equal probability to every outcome. Uniformity is the special case

[ p_k=\frac{1}{K} ]

for every (k). It also differs from a distribution over several simultaneous binary attributes, since a categorical trial permits exactly one category to be active.

Statistical estimation

For independent observations (x_1,\ldots,x_n), let (n_k) denote the number assigned to category (k). The likelihood function is

[ L(\boldsymbol{p})

\prod_{i=1}^{n}p_{x_i}

\prod_{k=1}^{K}p_k^{n_k}. ]

Its logarithm is

[ \ell(\boldsymbol{p})

\sum_{k=1}^{K}n_k\log p_k. ]

Under the simplex constraint, the maximum-likelihood estimator is

[ \widehat{p}_k=\frac{n_k}{n}. ]

This estimator is unbiased because

[ \operatorname{E}!\left[\frac{N_k}{n}\right]=p_k. ]

Its covariance is the categorical covariance matrix divided by (n), reflecting the information gained from repeated independent observations.

The likelihood-based treatment of class probabilities was incorporated into the general theory of statistical estimation by Ronald Fisher. Separately, Karl Pearson connected observed categorical counts with expected frequencies through the Pearson chi-squared statistic, which measures discrepancy between an empirical count vector and a specified categorical model.

With a Dirichlet prior, the posterior parameter vector is

[ \boldsymbol{p}\mid x_1,\ldots,x_n \sim \operatorname{Dirichlet} (\alpha_1+n_1,\ldots,\alpha_K+n_K). ]

The posterior mean of the (k)-th probability is

[ \operatorname{E}[p_k\mid x_1,\ldots,x_n]

\frac{\alpha_k+n_k} {\sum_{j=1}^{K}\alpha_j+n}. ]

This expression combines prior concentration with observed category counts in a single normalized parameter vector.

Information-theoretic quantities

The Shannon entropy of a categorical distribution is

[ H(\boldsymbol{p})

-\sum_{k=1}^{K}p_k\log p_k, ]

with the convention that (0\log 0=0). Entropy equals zero at every vertex of the simplex and reaches its maximum at the uniform distribution. With natural logarithms, the maximum value is (\log K).

For categorical parameter vectors (\boldsymbol{p}) and (\boldsymbol{q}), the cross-entropy is

[ H(\boldsymbol{p},\boldsymbol{q})

-\sum_{k=1}^{K}p_k\log q_k. ]

Their Kullback–Leibler divergence is

[ D_{\mathrm{KL}}(\boldsymbol{p}\Vert\boldsymbol{q})

\sum_{k=1}^{K} p_k\log\frac{p_k}{q_k}. ]

These quantities satisfy

[ H(\boldsymbol{p},\boldsymbol{q})

H(\boldsymbol{p}) + D_{\mathrm{KL}}(\boldsymbol{p}\Vert\boldsymbol{q}). ]

If (q_k=0) for a category having (p_k>0), both the cross-entropy and the divergence are infinite. This property reflects the incompatibility between assigning positive probability to an event and modeling that event as impossible.

Parameterization by logits

Categorical probabilities are frequently expressed through unconstrained real-valued parameters called logits. For a vector (\boldsymbol{\eta}\in\mathbb{R}^{K}), the softmax function defines

[ p_k

\frac{\exp(\eta_k)} {\sum_{j=1}^{K}\exp(\eta_j)}. ]

Adding the same constant to every logit leaves the probability vector unchanged. The parameterization is therefore non-identifiable unless one coordinate is fixed or another constraint is imposed.

In multinomial logistic regression, the logits depend on explanatory variables, and the resulting conditional distribution is categorical. In statistical classification, a model of the form

[ \Pr(Y=k\mid X=x)=p_k(x) ]

represents uncertainty over a single class label associated with an observed input. The negative log-likelihood of the observed class is identical to categorical cross-entropy.

Sampling structure

A categorical random variable admits an inverse-transform representation. Let (U) follow the continuous uniform distribution on ([0,1]), and define cumulative probabilities

[ c_k=\sum_{j=1}^{k}p_j. ]

The categorical outcome is the unique index (k) satisfying

[ c_{k-1}\leq U<c_k, ]

where (c_0=0). This construction partitions the unit interval into subintervals whose lengths equal the category probabilities.

An equivalent perturbation representation uses independent standard Gumbel distributions. If (G_1,\ldots,G_K) are such variables, then

[ X

\operatorname*{arg,max}_{k} \bigl(\log p_k+G_k\bigr) ]

has categorical probabilities (\boldsymbol{p}), provided categories with zero probability are excluded from the maximization. This identity connects categorical variables with random-utility models and differentiable relaxations of discrete optimization.

See also

Related articles include the multinomial distribution, the Bernoulli distribution, the Dirichlet distribution, the probability simplex, multinomial logistic regression, one-hot encoding, and entropy in information theory.