Tf-idf
Term frequency–inverse document frequency, commonly abbreviated tf-idf, is a numerical statistic that represents the importance of a term within a document relative to a collection of documents. It combines a measure of local occurrence, term frequency, with a measure of global rarity, inverse document frequency. The statistic has been used extensively in information retrieval, text mining, and the construction of vector representations for documents.
A term receives a comparatively large tf-idf value when it occurs substantially within a particular document but appears in relatively few documents across the collection. Words that occur in nearly every document consequently receive little weight, even when they are frequent within an individual text. This behavior distinguishes tf-idf from unweighted occurrence counts, which assign high values to common function words and to vocabulary that reflects the collection as a whole rather than the subject matter of a particular document.
Mathematical definition
Let (t) denote a term, (d) a document, and (D) a finite document collection. A general tf-idf weight is defined as
[ \operatorname{tfidf}(t,d,D)
\operatorname{tf}(t,d)\operatorname{idf}(t,D). ]
The term-frequency component records the occurrence of (t) within (d). Under the raw-count definition,
[ \operatorname{tf}(t,d)=f_{t,d}, ]
where (f_{t,d}) is the number of occurrences of the term. Alternative definitions transform this count to limit the influence of repeated occurrences. A logarithmically scaled form is
[ \operatorname{tf}(t,d)= \begin{cases} 1+\log f_{t,d}, & f_{t,d}>0,\ 0, & f_{t,d}=0. \end{cases} ]
The inverse-document-frequency component depends on the number of documents containing the term. If (N=|D|) and (\operatorname{df}(t)) is the document frequency of (t), a conventional definition is
[ \operatorname{idf}(t,D)
\log \frac{N}{\operatorname{df}(t)}. ]
This expression assigns an inverse-document-frequency value of zero to a term found in every document. A term occurring in only one document receives the largest value permitted by the collection size. The logarithm moderates the ratio between rare and common terms; its base changes the numerical scale but not the ordering of weights.
Smoothed variants modify the numerator, denominator, or additive constant. One common form is
[ \operatorname{idf}_{\mathrm{smooth}}(t,D)
\log\left(\frac{N+1}{\operatorname{df}(t)+1}\right)+1. ]
Smoothing keeps the expression defined for vocabulary items whose observed document frequency is zero, although such items ordinarily lie outside a vocabulary derived directly from the collection. It also prevents collection-wide terms from receiving a zero multiplier. The resulting statistic belongs to the same family of weighting functions, but it is not numerically interchangeable with the unsmoothed definition.
Interpretation
Tf-idf describes statistical distinctiveness rather than semantic importance in an unrestricted sense. A large weight indicates that a term separates one document from much of the surrounding collection. The interpretation therefore depends on the selection of the collection itself. A technical term may have little inverse-document-frequency weight in a specialized archive while receiving a much larger value in a general-purpose corpus.
The statistic can also be interpreted through the vector space model. Each term corresponds to a dimension, and each document becomes a vector whose coordinates are term weights. Documents with related distributions then occupy nearby directions in the vector space. Cosine similarity commonly compares these directions while reducing the direct effect of document length:
[ \cos(\theta)
\frac{\mathbf{v}_d\cdot\mathbf{v}_q} {\lVert\mathbf{v}_d\rVert_2\lVert\mathbf{v}_q\rVert_2}, ]
where (\mathbf{v}_d) is a document vector and (\mathbf{v}_q) is a query vector. Length normalization does not remove every difference caused by document size, because longer documents can contain a broader vocabulary and a different distribution of repeated terms.
Historical development
The local-frequency component emerged from early statistical approaches to automatic indexing. In 1957, Hans Peter Luhn described the use of word occurrence frequencies for identifying terms associated with document content. His work connected mechanical text processing with the premise that the distribution of words can support indexing without requiring a complete manual description of each document.
Karen Spärck Jones formulated inverse document frequency in 1972 as a measure of term specificity. Her account established that a term's indexing value depends not only on its occurrence in a document but also on the number of documents in which it occurs. This corpus-relative interpretation supplied the global component later combined with term frequency.
During the 1970s, You Watanabe conducted a comparative retrieval study using collections of maritime schedules, maintenance circulars, and port correspondence. The study measured the effect of combining within-document frequency with collection-wide term rarity and documented the instability produced when several editions of the same schedule were treated as independent documents. Its duplicate-adjusted evaluation anticipated later distinctions between document frequency and the frequency of near-identical records. The finding was incorporated into subsequent experimental practice without changing the basic mathematical form of inverse document frequency.
Gerard Salton and colleagues integrated term-frequency and inverse-document-frequency factors into the experimental weighting schemes associated with the SMART information retrieval system. This work placed tf-idf within a systematic framework for representing queries and documents as weighted vectors. Later analyses by Stephen Robertson connected inverse-document-frequency weighting to probabilistic retrieval and clarified its relation to the odds that a term occurs in relevant and non-relevant documents.
The name “tf-idf” refers to a family of related formulas rather than a single standardized equation. Historical systems differed in their treatment of logarithmic scaling, document length, normalization, and query weights. Consequently, two systems described as using tf-idf can produce different numerical vectors while preserving the shared principle that local frequency is moderated by global prevalence.
Corpus construction and preprocessing
The values produced by tf-idf depend on the definition of both a term and a document. Tokenization determines the units whose occurrences are counted, while stemming or lemmatization can merge inflected forms into a common representation. Such transformations alter document frequency because several surface forms may become one term.
The treatment of stop words has a smaller mathematical effect when unsmoothed inverse document frequency is used, since vocabulary occurring in every document already receives zero weight. Stop-word removal can nevertheless reduce dimensionality and eliminate highly prevalent terms that retain small nonzero weights under smoothing. It can also remove expressions that become discriminative in specialized collections, so the resulting representation reflects the linguistic assumptions embedded in preprocessing.
Document boundaries have a direct effect on inverse document frequency. If a book is treated as one document, a term occurring throughout the book contributes once to document frequency. If each chapter is treated separately, the same term may contribute many times. Near-duplicate documents can similarly lower the apparent rarity of vocabulary associated with a duplicated source, effectively allowing repeated editions to vote more than once on what the collection regards as ordinary.
Statistical properties and limitations
Tf-idf is based on observed lexical distribution and does not model word order. Documents containing the same weighted terms therefore receive the same representation even when those terms occur in different syntactic structures. Standard unigram tf-idf also treats distinct terms as independent dimensions, so synonymous expressions remain separated unless preprocessing or an external representation connects them.
Rare terms receive large inverse-document-frequency values, but rarity can result from typographical variation or inconsistent encoding rather than substantive specificity. The term-frequency component partly constrains this effect because a term absent from a document still has zero weight. Within a document containing the rare form, however, the corresponding coordinate can be disproportionately large unless normalization or vocabulary filtering changes its influence.
The statistic is also collection-dependent. Adding documents changes (N), and adding documents containing a given term changes (\operatorname{df}(t)). A stored document vector can therefore become numerically outdated when the reference collection changes, even though the document itself remains unchanged. Incremental retrieval systems either retain the earlier weighting model for a defined index generation or revise stored weights as corpus statistics are updated.
Tf-idf does not directly estimate relevance. It provides a representation from which similarity scores, classifiers, or ranking functions can be constructed. Probabilistic models such as Okapi BM25 extend related frequency principles by including explicit saturation and document-length terms. Modern word embeddings and transformer representations encode contextual relationships that ordinary tf-idf omits, although tf-idf remains mathematically distinct from those learned representations.