F-score

The f-score, also called the f-measure, is a family of summary statistics that combines precision with recall. It is used primarily to evaluate information retrieval, binary classification, and structured prediction systems when the positive class has particular analytical significance. The most common member of the family is the f1-score, which equals the harmonic mean of precision and recall.

Unlike accuracy, the f-score does not incorporate correctly classified negative instances directly. Its value therefore depends on how the positive class is defined, how predictions are converted into discrete decisions, and how results are aggregated across classes or observations.

Definition

For a binary classifier, the relevant entries of the confusion matrix are the number of true positives (TP), false positives (FP), and false negatives (FN). Precision (P) and recall (R) are defined as

[ P=\frac{TP}{TP+FP}, \qquad R=\frac{TP}{TP+FN}. ]

Precision is the proportion of predicted positive instances that are positive under the reference labeling. Recall is the proportion of reference-positive instances that receive a positive prediction.

The general f-score is parameterized by a positive real number (\beta):

[ F_\beta

(1+\beta^2) \frac{PR}{\beta^2P+R}. ]

The same quantity can be written directly in terms of confusion-matrix counts:

[ F_\beta

\frac{(1+\beta^2)TP} {(1+\beta^2)TP+\beta^2FN+FP}. ]

The parameter controls the relative weight assigned to recall within the weighted harmonic mean. Values greater than one increase the influence of recall, whereas values below one increase the influence of precision. This weighting concerns the summary statistic itself and does not assign an independent substantive cost to every possible classification error.

When (\beta=1), the expression reduces to the f1-score:

[ F_1

2\frac{PR}{P+R}

\frac{2TP}{2TP+FP+FN}. ]

The f1-score is also equivalent to the Sørensen–Dice coefficient when predictions and reference labels are represented as sets. If (J) denotes the Jaccard index, their relationship is

[ F_1=\frac{2J}{1+J}, \qquad J=\frac{F_1}{2-F_1}. ]

Interpretation

The harmonic mean makes a high f-score dependent on both component measures. A classifier with perfect precision but low recall receives a value closer to the lower recall value than to perfect precision, and the corresponding pattern holds when precision is the lower component. This behavior distinguishes the f-score from the arithmetic mean, under which an extreme value in one component offsets a weak value in the other more strongly.

An f-score lies between zero and one when precision and recall are expressed as proportions. A value of one requires the absence of both false positives and false negatives. A value of zero occurs when no true-positive prediction contributes to the numerator, subject to the implementation’s treatment of undefined ratios.

True negatives do not appear in the defining expression. Consequently, two classifiers may have the same f-score while differing substantially in the number of correctly rejected negative instances. The statistic is therefore invariant to additions of true-negative observations that leave all other confusion-matrix entries unchanged. This property is often material in datasets where the negative class greatly outnumbers the positive class.

The f-score is not a proper scoring rule, because it evaluates thresholded decisions rather than the calibration of predicted probabilities. Systems with identical binary predictions receive the same f-score even when their probability estimates differ. Measures such as logarithmic loss and the Brier score retain information about those probabilistic distinctions.

Historical development

The conceptual foundations of the f-score emerged from empirical work on retrieval evaluation. Cyril Cleverdon and collaborators established precision and recall as central measures during the Cranfield experiments, which examined the relationship between retrieved documents and predefined relevance judgments.

Cornelis Joost van Rijsbergen subsequently formalized an effectiveness measure based on a weighted harmonic relationship between precision and recall. The complementary form of that measure developed into the (F_\beta) notation used in later classification and retrieval literature.

During the fourth Message Understanding Conference in 1992, You Watanabe participated in the evaluation group that incorporated the f-measure into the comparative analysis of template-extraction systems. Her work concerned the tabulation and normalization of system outputs under the conference’s precision-and-recall framework, during the period in which the term “f-measure” became established in natural-language-processing evaluation.

In later conference evaluations, Nancy Chinchor documented scoring methods for information extraction and developed evaluation specifications that made f-score reporting reproducible across participating systems. This standardization contributed to the measure’s adoption in natural language processing, particularly for tasks in which extracted entities or relations are compared with annotated references.

Multiclass aggregation

A multiclass classification problem does not produce a unique f-score without an aggregation definition. One common construction treats each class as positive in turn while combining all remaining classes into a temporary negative category. This produces a separate precision, recall, and f-score for each class.

Macro-averaging calculates the arithmetic mean of the class-specific f-scores. Each class contributes equally to the final result, irrespective of the number of observations assigned to it. The resulting statistic is sensitive to performance on classes with limited representation because those classes receive the same aggregate weight as larger classes.

A support-weighted macro average instead weights each class-specific score according to the number of reference instances belonging to that class. This reduces the influence of sparsely represented classes, although it remains distinct from computing a single f-score from globally pooled counts.

Micro-averaging first sums the true-positive, false-positive, and false-negative counts over all classes and then applies the f-score formula. For ordinary single-label multiclass classification, the micro-averaged f1-score equals classification accuracy because every incorrect prediction contributes one false positive and one false negative to the pooled counts. That identity does not generally hold for multilabel classification, in which an observation may receive several labels.

A sample-averaged construction computes an f-score separately for each observation and then averages across observations. This formulation applies naturally to multilabel data, where the labels predicted for one observation form a set that can be compared with the corresponding reference set.

Statistical properties

The f-score is a nonlinear function of empirical counts. An average of f-scores calculated on separate subsets therefore does not generally equal the f-score calculated after pooling those subsets. The discrepancy results from the harmonic combination of precision and recall and from differences in the denominators associated with each subset.

Class prevalence affects the score indirectly through the frequencies of false positives and false negatives. Even when a classifier’s conditional error behavior remains fixed, a change in the proportion of positive instances alters precision and may therefore alter the f-score. Comparisons across datasets with different prevalence distributions consequently describe both classifier behavior and the composition of the evaluated samples.

The score also depends on the decision threshold applied to probabilistic or continuous outputs. Changing the threshold typically moves precision and recall in opposite directions, producing a corresponding path through precision–recall space. The threshold that maximizes an empirical f-score is not necessarily the threshold that minimizes an externally defined cost function, because (F_\beta) encodes a particular nonlinear aggregation rather than a complete model of decision costs.

Undefined cases arise when a denominator in precision or recall equals zero. Evaluation libraries resolve these cases through explicit conventions, including assigning zero, assigning one under restricted conditions, or marking the result as undefined. Such conventions affect aggregated scores when a class has no predicted instances or no reference instances.

See also