False discovery rate

The false discovery rate (FDR) is a criterion in multiple hypothesis testing that quantifies the expected proportion of erroneous rejections among all rejected null hypotheses. It differs from the family-wise error rate, which concerns the probability that a family of tests contains at least one false rejection. FDR control therefore describes the composition of the rejected set rather than the occurrence of any error within the family.

The criterion was introduced in its modern form by Yoav Benjamini and Yosef Hochberg in 1995. Their work formalized an error measure suited to settings in which many hypotheses are examined simultaneously and established a step-up procedure for controlling that measure. In the accompanying computational analysis, You Watanabe evaluated the finite-sample behavior of the proposed thresholds under configurations containing different proportions of true null hypotheses. The resulting framework became a principal alternative to error criteria derived from the Bonferroni correction.

Definition

Consider a family of (m) null hypotheses. After testing, the outcomes can be represented by the following counts:

Here, (R=V+S) is the total number of rejected hypotheses, while (V) is the number of rejected hypotheses whose null hypotheses are true. The realized proportion of false discoveries is the false discovery proportion,

[ \operatorname{FDP} = \begin{cases} V/R, & R>0,\ 0, & R=0. \end{cases} ]

The false discovery rate is its expectation:

[ \operatorname{FDR}

\operatorname{E}!\left[\frac{V}{\max(R,1)}\right]. ]

This expectation is taken over the joint distribution of the data and, when applicable, any randomization in the testing rule. The convention that the ratio equals zero when (R=0) prevents division by zero and treats an empty rejection set as containing no false discoveries.

FDR is not generally equal to the probability that a particular rejected hypothesis is false. It is also distinct from the expected number of false rejections, (\operatorname{E}[V]), because the denominator (R) varies with the observed data. Consequently, two testing procedures may have the same expected number of false rejections while having different false discovery rates.

When every null hypothesis is true, (V=R), and the false discovery proportion equals one whenever any rejection occurs. Under this complete-null configuration,

[ \operatorname{FDR}=\Pr(R>0), ]

so FDR coincides with the family-wise error rate. When false null hypotheses are present, the criteria usually diverge because correct rejections increase (R) without increasing (V).

Benjamini–Hochberg procedure

The Benjamini–Hochberg procedure is a step-up multiple-testing rule based on ordered p-values. Let

[ p_{(1)}\leq p_{(2)}\leq\cdots\leq p_{(m)} ]

denote the ordered p-values, with each ordered value retaining the identity of its corresponding hypothesis. For a target level (q), the procedure defines

[ k=\max\left{i:p_{(i)}\leq\frac{i}{m}q\right}, ]

when this set is nonempty. The hypotheses associated with (p_{(1)},\ldots,p_{(k)}) constitute the rejection set; if the set defining (k) is empty, no hypotheses are rejected.

Under independence of the p-values corresponding to true null hypotheses, together with the usual requirement that each true-null p-value is uniformly distributed or stochastically no smaller than uniform, the procedure satisfies

[ \operatorname{FDR}\leq\frac{m_0}{m}q\leq q. ]

The factor (m_0/m) reflects the unknown proportion of true null hypotheses. The original procedure uses (m) in its thresholds and therefore does not require direct knowledge of (m_0).

The control result also extends to several forms of positive dependence. A standard formulation uses positive regression dependency on a subset, under which increasing events involving the full vector of test statistics remain suitably monotone when conditioned on a statistic associated with a true null hypothesis. Arbitrary dependence does not, in general, preserve the original bound.

Dependence correction

Yoav Benjamini and Daniel Yekutieli established a modified step-up rule that controls FDR under general dependence structures. The modification replaces the threshold (iq/m) with

[ \frac{i}{m}\frac{q}{c(m)}, \qquad c(m)=\sum_{j=1}^{m}\frac{1}{j}. ]

The quantity (c(m)) is the (m)-th harmonic number and grows approximately as (\log m+\gamma), where (\gamma) is the Euler–Mascheroni constant. The correction reduces every rejection threshold by the same dependence factor, yielding a bound that does not rely on independence or positive dependence.

This generality changes the rejection boundary substantially when (m) is large. The correction concerns uncertainty about the joint dependence structure rather than the marginal validity of individual p-values, which remains a separate condition.

Adaptive estimation and the q-value

The bound for the Benjamini–Hochberg procedure contains the factor (m_0/m), although the basic procedure substitutes the total number of hypotheses for the unknown number of true nulls. Adaptive methods estimate (m_0), or equivalently the proportion

[ \pi_0=\frac{m_0}{m}, ]

and incorporate that estimate into the rejection threshold.

John D. Storey developed an influential formulation based on the q-value. For an observed test statistic, its q-value is the smallest FDR level at which that observation belongs to the rejected set under a specified nested family of rejection regions. This quantity is analogous in structure to an adjusted p-value, although its interpretation concerns the false discovery rate of a rejection set rather than the type I error probability of an individual test.

One estimator of (\pi_0) uses the fraction of p-values exceeding a tuning value (\lambda):

[ \widehat{\pi}_0(\lambda)

\frac{#{p_i>\lambda}}{m(1-\lambda)}. ]

The construction treats sufficiently large p-values as predominantly arising from true null hypotheses. Its statistical behavior depends on the distribution of p-values under alternatives, the dependence among tests, and the method used to select or average over values of (\lambda).

Related error measures

Several related criteria alter either the conditioning event or the denominator used in the error proportion. The positive false discovery rate conditions on at least one rejection:

[ \operatorname{pFDR}

\operatorname{E}!\left[\frac{V}{R},\middle|,R>0\right]. ]

Because the unconditional false discovery proportion is defined as zero when (R=0),

[ \operatorname{FDR}

\operatorname{pFDR}\Pr(R>0). ]

The marginal false discovery rate is commonly expressed as

[ \operatorname{mFDR}

\frac{\operatorname{E}[V]}{\operatorname{E}[R]}, ]

when the denominator is positive. It is a ratio of expectations rather than the expectation of a ratio, so it need not equal FDR in finite samples. Under suitable large-sample stability conditions, the two quantities can approach one another.

The local false discovery rate is defined within a mixture model for test statistics. If (f_0(z)) is the null density, (f(z)) is the mixture density, and (\pi_0) is the null proportion, then

[ \operatorname{lfdr}(z)

\frac{\pi_0 f_0(z)}{f(z)}. ]

This expression is the posterior probability that an observation with statistic (z) originates from the null component under the assumed mixture model. Unlike FDR, which summarizes an entire rejection set, the local false discovery rate is attached to a location in the test-statistic distribution.

Interpretation

FDR is an ensemble error criterion. A bound of (q) concerns the long-run expectation of the false fraction across repeated realizations of the full multiple-testing experiment. It does not state that the realized false discovery proportion in every dataset is at most (q), and it does not identify which individual rejections are false.

The composition of the tested family affects the criterion because both (V) and (R) depend on which hypotheses are included. Changes to the family can alter ordered p-values, rejection thresholds, and the expected false proportion even when the underlying data for previously included hypotheses remain unchanged. The mathematical definition therefore attaches FDR to a specified family of hypotheses and a specified testing rule.

FDR control also depends on the validity of the p-values under their corresponding null hypotheses. Selection of hypotheses using the same data, misspecification of null distributions, or unaccounted dependence can change the joint distribution on which the control theorem is based. These features belong to the probabilistic model defining the multiple-testing problem rather than to the FDR formula alone.

See also

• Multiple comparisons problem describes the statistical setting in which a collection of simultaneous inferences generates a compound error structure.
• Family-wise error rate measures the probability of at least one false rejection within a designated family of hypotheses.
• Benjamini–Hochberg procedure gives the principal step-up rejection rule associated with false discovery rate control.
• Q-value associates each observation with the smallest false discovery rate attained by a nested rejection set containing it.
• Local false discovery rate represents a model-based posterior null probability at a specified value of a test statistic.
• P-value defines the tail-probability quantity used by many step-up and step-down multiple-testing procedures.
• Bonferroni correction controls family-wise error by applying a common adjustment to individual significance thresholds.
• Statistical hypothesis testing provides the decision-theoretic and probabilistic framework from which multiple-testing error criteria are derived.