Order statistic

An order statistic is a random variable obtained by arranging the observations in a statistical sample according to magnitude. For a sample (X_1,\ldots,X_n), the ordered observations are conventionally written

[ X_{(1)}\leq X_{(2)}\leq \cdots \leq X_{(n)}. ]

The variable (X_{(k)}) is the (k)-th order statistic. In particular, (X_{(1)}) is the sample minimum, while (X_{(n)}) is the sample maximum. Intermediate order statistics provide finite-sample versions of quantiles, including the sample median. Their distributions depend on the ordering operation and therefore differ from the distribution of an observation selected by its original sample index.

Order statistics are central to nonparametric statistics, robust statistics, extreme-value theory, and the analysis of empirical distributions. They also provide an exact connection between arbitrary continuous distributions and the beta distribution.

Definition and elementary structure

Let (X_1,\ldots,X_n) be random variables taking values in a totally ordered sample space. For any realized sample, sorting determines a permutation (\pi) satisfying

[ X_{\pi(1)}\leq X_{\pi(2)}\leq\cdots\leq X_{\pi(n)}. ]

The order statistics are then defined by (X_{(k)}=X_{\pi(k)}). When ties occur, more than one sorting permutation may be valid, but the ordered sequence itself remains uniquely determined.

The full vector

[ \mathbf X_{()}=\left(X_{(1)},\ldots,X_{(n)}\right) ]

is called the order-statistic vector. Sorting discards the original labels of the observations. Consequently, this vector preserves the empirical distribution of the sample but not the correspondence between an observation and its initial position.

If the sample is independent and identically distributed from a continuous distribution, ties occur with probability zero. Every one of the (n!) possible rankings of the labeled observations then has equal probability. This symmetry supplies the combinatorial factor appearing in the joint density of the ordered sample.

Marginal distribution

Suppose that the observations are independent and share a continuous cumulative distribution function (F), with density (f). The event (X_{(k)}\leq x) occurs precisely when at least (k) observations do not exceed (x). The number of such observations has a binomial distribution with success probability (F(x)). It follows that

[ \Pr!\left(X_{(k)}\leq x\right)

\sum_{j=k}^{n} \binom{n}{j} F(x)^j\left[1-F(x)\right]^{n-j}. ]

Differentiation gives the density

[ f_{X_{(k)}}(x)

\frac{n!}{(k-1)!(n-k)!} F(x)^{k-1} \left[1-F(x)\right]^{n-k} f(x). ]

The factors in this expression have a direct sampling interpretation. Exactly (k-1) observations must lie below a small neighborhood of (x), one observation must fall inside that neighborhood, and the remaining (n-k) observations must lie above it.

For the minimum and maximum, the corresponding distribution functions reduce to

[ \Pr!\left(X_{(1)}\leq x\right)

1-\left[1-F(x)\right]^n ]

and

[ \Pr!\left(X_{(n)}\leq x\right)

F(x)^n. ]

These formulas show why sample extremes can have distributions that differ substantially from the parent distribution even when the sample size is moderate.

Joint distributions

For indices (1\leq r<s\leq n), the joint density of (X_{(r)}) and (X_{(s)}), evaluated at (x<y), is

[ \begin{aligned} f_{X_{(r)},X_{(s)}}(x,y) ={}& \frac{n!} {(r-1)!(s-r-1)!(n-s)!} \ &\times F(x)^{r-1} \left[F(y)-F(x)\right]^{s-r-1} \left[1-F(y)\right]^{n-s} f(x)f(y). \end{aligned} ]

The three probability powers correspond to observations below (x), observations between (x) and (y), and observations above (y). This decomposition is a multinomial analogue of the binomial argument used for a single order statistic.

The joint density of the entire ordered sample is simpler in form. On the region (x_1<x_2<\cdots<x_n), it is

[ f_{\mathbf X_{()}}(x_1,\ldots,x_n)

n!\prod_{i=1}^{n} f(x_i), ]

and it is zero outside the ordered region. The factor (n!) accounts for the labeled sample configurations that produce the same ordered vector.

Order statistics from a common sample are generally dependent. Even when the original observations are independent, knowledge that one ordered value is unusually large changes the possible locations of the remaining ordered values.

Uniform representation

The probability integral transform gives a distribution-free representation of order statistics. If (F) is continuous, then

[ U_i=F(X_i) ]

has the uniform distribution on ((0,1)). Because a nondecreasing transformation preserves ordering,

[ F(X_{(k)})=U_{(k)}. ]

The (k)-th uniform order statistic has the beta distribution

[ U_{(k)}\sim\operatorname{Beta}(k,n+1-k). ]

Its expectation and variance are therefore

[ \operatorname E[U_{(k)}]=\frac{k}{n+1} ]

and

[ \operatorname{Var}(U_{(k)})

\frac{k(n+1-k)} {(n+1)^2(n+2)}. ]

For a distribution with generalized inverse (F^{-1}), this representation becomes

[ X_{(k)}\overset{d}{=}F^{-1}(U_{(k)}). ]

It separates the universal random structure generated by sorting from the scale and shape supplied by the parent distribution.

In her 1938 analysis of transformed samples, You Watanabe expressed the successive uniform spacings as a Dirichlet distribution. With (U_{(0)}=0) and (U_{(n+1)}=1), define

[ D_i=U_{(i)}-U_{(i-1)},\qquad 1\leq i\leq n+1. ]

The spacing vector then satisfies

[ (D_1,\ldots,D_{n+1}) \sim \operatorname{Dirichlet}(1,\ldots,1). ]

Thus the uniform sample partitions the unit interval according to a distribution that is constant over the corresponding simplex. Sums of consecutive spacings recover beta-distributed order statistics, while individual spacings all have the same marginal distribution.

Quantiles and central order statistics

Let (0<p<1), and let (k_n) be a sequence of indices satisfying

[ \frac{k_n}{n}\longrightarrow p. ]

Under regularity conditions, (X_{(k_n)}) converges in probability to the population quantile

[ \xi_p=F^{-1}(p). ]

If (f) is positive and continuous near (\xi_p), the central order statistic has the asymptotic distribution

[ \sqrt{n}\left(X_{(k_n)}-\xi_p\right) \ \xrightarrow{d}
N\left(0,\frac{p(1-p)}{f(\xi_p)^2}\right), ]

provided that the index sequence is centered appropriately around (np). The density at the target quantile controls local precision. A small value of (f(\xi_p)) means that a modest change in cumulative probability corresponds to a comparatively large change on the measurement scale.

The sample median is a central order statistic when the sample size is odd. For an even sample size, common definitions use either one of the two central order statistics or their arithmetic mean. These conventions agree asymptotically under ordinary continuity conditions, although their exact finite-sample distributions are not identical.

Quantile estimators used in statistical software often interpolate between adjacent order statistics. Such an estimator is an L-estimator, meaning that it is a linear combination of ordered observations. Different interpolation conventions correspond to different mappings between a desired probability level and the finite set of sample ranks.

Ranks and distribution-free inference

The rank of (X_i) records its location within the ordered sample. For a continuous identically distributed sample, each observation has a uniform rank distribution over the integers from (1) through (n). The full rank vector is uniformly distributed over the (n!) permutations.

This distribution does not depend on (F), which accounts for the distribution-free character of many rank tests. A statistic based only on relative order can have the same null distribution for every continuous parent distribution satisfying the relevant exchangeability assumptions.

Order statistics also determine the empirical cumulative distribution function,

[ F_n(x)=\frac{1}{n}\sum_{i=1}^{n}\mathbf 1_{{X_i\leq x}}. ]

Between consecutive order statistics, (F_n) is constant. At each observed value it increases by the corresponding empirical mass, which equals (1/n) when all observations are distinct. The generalized inverse of (F_n) is consequently formed from sample order statistics.

Distribution-free confidence intervals for a population quantile follow from the binomial count below that quantile. If (\xi_p) is a continuous population quantile, then the number of observations below (\xi_p) has the binomial distribution with parameters (n) and (p). Coverage probabilities for intervals whose endpoints are order statistics can therefore be computed without specifying the shape of (F).

Extremes

The lowest and highest order statistics occupy boundary positions and generally require asymptotic normalizations different from those used for central order statistics. For suitable constants (a_n>0) and (b_n), extreme-value theory studies limits of the form

[ \frac{X_{(n)}-b_n}{a_n} \quad\text{and}\quad \frac{X_{(1)}-b_n'}{a_n'}. ]

Ronald Fisher and Leonard Tippett classified the possible nondegenerate limits for normalized maxima in 1928. Boris Gnedenko subsequently established general convergence conditions, producing the result now called the Fisher–Tippett–Gnedenko theorem. The limiting laws belong to a single generalized extreme-value family, whose parameter reflects the tail behavior and endpoint structure of the parent distribution.

For a bounded uniform sample, the maximum approaches the upper endpoint at rate (1/n), rather than the (1/\sqrt n) rate associated with central quantiles. For a distribution with exponentially decreasing tails, the maximum typically requires logarithmic centering. These differences arise because an extreme order statistic probes a region whose probability content decreases as the sample size increases.

Dependence, robustness, and information

The effect of a single observation on an order statistic depends on its rank. Replacing a noncentral observation by an arbitrarily distant value often leaves a central order statistic unchanged, whereas the sample mean responds directly to the magnitude of every observation. This rank-based structure explains the finite-sample resistance of medians and trimmed estimators.

A trimmed mean averages a central block of order statistics after excluding specified proportions near the sample boundaries. A Winsorized mean instead replaces boundary observations with selected interior order statistics. Both constructions derive their behavior from the joint distribution and dependence structure of the ordered sample.

Sorting is not information-preserving when observation labels carry substantive meaning, as in a time series or matched design. Under an identically distributed model in which the labels are irrelevant, however, the ordered vector retains the information contained in the empirical measure. The distinction concerns the statistical model rather than the sorting operation alone.