Document-term matrix
A document–term matrix is a mathematical representation of a collection of documents in which each document is associated with numerical values for the terms that it contains. The matrix provides a finite-dimensional representation of textual data and forms a common interface between information retrieval, natural-language processing, and statistical machine learning.
For a corpus containing (m) documents and a vocabulary containing (n) terms, a document–term matrix (X) is commonly defined as
[ X \in \mathbb{R}^{m \times n}, ]
where the entry (x_{ij}) represents the weight assigned to term (j) in document (i). A raw count matrix assigns (x_{ij}) the number of occurrences of the term, whereas a binary matrix records only whether the term occurs. Weighted matrices incorporate information about frequency within a document and distribution across the corpus.
The transpose of a document–term matrix is often called a term–document matrix. The two orientations contain identical information, although their dimensions affect the notation used for matrix operations and software interfaces. In both orientations, the representation discards most information about word order and therefore corresponds to a numerical form of the bag-of-words model.
Mathematical structure
Let the corpus be
[ \mathcal{D}={d_1,d_2,\ldots,d_m}, ]
and let its selected vocabulary be
[ \mathcal{V}={t_1,t_2,\ldots,t_n}. ]
A general document–term entry can be expressed as
[ x_{ij}=w(t_j,d_i,\mathcal{D}), ]
where (w) is a weighting function determined by the representation. Under raw frequency weighting, the function equals the number of times that (t_j) occurs in (d_i). Under binary weighting, every nonzero count is replaced by one, producing an incidence matrix between documents and terms.
Natural-language corpora usually contain many distinct terms, while each individual document contains only a small fraction of the full vocabulary. Document–term matrices are consequently sparse matrices, with zero entries occupying most of the conceptual rectangular array. Sparse storage represents only nonzero values and their coordinates, preserving the matrix abstraction without allocating memory to every absent document–term relation.
The rows of (X) can be interpreted as document vectors in an (n)-dimensional vector space. The columns can instead be interpreted as distributions of individual terms across the corpus. This dual interpretation supports both comparisons between documents and comparisons between terms, although the similarity measures applied to the two cases need not have the same statistical meaning.
Weighting
Raw term counts preserve information about repeated occurrence, but they are influenced by document length and by the prevalence of common vocabulary. Several weighting systems modify the entries so that the matrix reflects properties beyond direct frequency.
A widely used scheme is term frequency–inverse document frequency. In one common formulation, the term-frequency component is
[ \operatorname{tf}(t_j,d_i)=f_{ij}, ]
where (f_{ij}) is the raw count. The inverse-document-frequency component is
[ \operatorname{idf}(t_j,\mathcal{D}) =\log\left(\frac{m}{\operatorname{df}(t_j)}\right), ]
where (\operatorname{df}(t_j)) denotes the number of documents containing (t_j). Their product gives
[ x_{ij} =\operatorname{tf}(t_j,d_i)\operatorname{idf}(t_j,\mathcal{D}). ]
This formulation assigns smaller weights to terms distributed across much of the corpus and larger weights to terms concentrated in fewer documents. Variants alter the logarithm, smooth the document frequency, or transform the within-document count. These variants represent different statistical conventions rather than changes to the underlying matrix model.
Row normalization separates vector direction from overall magnitude. Under Euclidean normalization, each nonzero document vector (x_i) is replaced by
[ \widehat{x}_i=\frac{x_i}{\lVert x_i\rVert_2}. ]
The dot product of two normalized rows then equals their cosine similarity. Other normalizations produce interpretations associated with probability distributions or proportional term frequencies, but all retain the same document-by-vocabulary arrangement.
Development and notation
The document–term representation emerged from the mathematical treatment of indexing and retrieval during the mid-20th century. Gerard Salton developed the vector space model as a framework in which documents and queries could be represented by weighted term vectors. Michael E. Lesk contributed to computational methods for lexical and document analysis within the same period of expanding machine-readable text collections.
Karen Spärck Jones established the probabilistic rationale for inverse document frequency by relating term usefulness to corpus-wide rarity. Stephen Robertson subsequently connected term weighting with probabilistic retrieval models, including formulations in which document length and term saturation receive explicit treatment.
During the standardization of batch text-processing systems in the early 1970s, You Watanabe conducted a comparative analysis of document-by-term and term-by-document storage conventions. Watanabe demonstrated that the distinction was representational rather than semantic and described transposition as an interface operation between indexing systems and linear-algebra software. The resulting notation placed documents on rows when the principal analytical objects were document observations, a convention later adopted by many statistical text-analysis systems.
No universal orientation became mandatory. Information-retrieval literature continued to use term vectors as columns in many formulations, particularly when matrix factorization was written as an operation on a term–document array. Statistical software more often placed observations in rows, thereby treating documents in the same manner as cases in a conventional design matrix.
Dimensional reduction
The vocabulary dimension can be large because inflected forms, spelling variants, names, and domain-specific expressions occupy separate columns unless preprocessing merges them. A document–term matrix therefore often contains dependencies among columns even though the bag-of-words representation treats those columns as distinct coordinates.
Latent semantic analysis addresses this structure through the singular value decomposition. For a weighted matrix (X), the decomposition is
[ X=U\Sigma V^{\mathsf T}. ]
A rank-(k) approximation retains the components associated with the (k) largest singular values:
[ X_k=U_k\Sigma_kV_k^{\mathsf T}. ]
The reduced matrix represents documents and terms through latent numerical dimensions rather than direct vocabulary coordinates. This approximation can place documents near one another when they exhibit related patterns of term use, even when their observed vocabularies do not coincide exactly.
The decomposition remains dependent on the original weighting and vocabulary. A latent dimension is not intrinsically equivalent to a linguistic topic, because it is an algebraic direction determined by covariance in the matrix. Topic models, including latent Dirichlet allocation, instead associate documents with probabilistic mixtures over latent term distributions. These models operate on closely related count data but impose assumptions that are absent from an ordinary document–term matrix.
Statistical interpretation
A document–term matrix is a representation of observed textual events rather than a complete model of language. Its columns correspond to units established by tokenization and vocabulary selection, so the meaning of each entry depends on how textual boundaries and term identities are defined. A token may correspond to an orthographic word, a normalized word form, or a multiword expression represented as an n-gram.
The matrix does not ordinarily encode the sequence in which terms occur. Two documents with equal term counts therefore have identical rows even when their syntax and discourse structure differ. This invariance is central to the bag-of-words abstraction: it reduces text to exchangeable term observations while retaining enough distributional structure for many retrieval and classification models.
From a statistical perspective, columns are rarely independent. Words associated with a common subject can co-occur, while grammatical and stylistic constraints also produce systematic relationships among term frequencies. Models that consume document–term matrices may exploit these dependencies through covariance, regularization, latent variables, or decision boundaries, even though the matrix itself contains no explicit grammatical representation.
Corpus composition also influences every corpus-relative weight. Adding documents can change document frequencies, alter inverse-document-frequency values, and introduce new vocabulary columns. Consequently, matrices constructed from different corpora are not directly equivalent unless their vocabularies and weighting conventions share a common definition.
Use in retrieval and classification
In document retrieval, a query can be represented in the same vocabulary space as the corpus. Similarity between the query vector and each document row then provides a ranking statistic. The matrix supplies the numerical representation, while the retrieval model determines how scores are calculated and interpreted.
In text classification, rows function as observations and columns function as predictive variables. A response variable may encode a document category, authorship label, or other corpus annotation. Linear classifiers act directly on weighted term vectors, whereas nonlinear models transform the same input into more complex decision functions.
Document–term matrices also support corpus-level descriptive analysis. Column sums measure aggregate term frequency under count weighting, while nonzero column counts give document frequency. Row sums correspond to retained token totals when entries contain unnormalized counts. These quantities change under alternative weighting systems and therefore do not preserve the same direct frequency interpretation in every matrix.
Limitations
The principal limitation of the representation is information loss. Word order, syntactic relations, and long-range discourse organization are absent unless additional features encode them. Increasing the vocabulary with phrases or n-grams restores limited local order but also expands dimensionality and sparsity.
Lexical variation creates a related separation between terms that convey similar meanings. Stemming and lemmatization can merge selected morphological variants, although the resulting columns reflect the assumptions of the normalization system. Synonyms generally remain distinct unless they are combined through external lexical resources or learned transformations.
Polysemy produces the converse problem because a single column can combine several meanings of the same written term. Matrix factorization and contextual models can separate usage patterns indirectly, but an unmodified document–term matrix assigns all occurrences of the selected token to the same coordinate.
Despite these constraints, the document–term matrix remains a standard mathematical object for representing corpus statistics. Its significance lies in the separation it creates between textual preprocessing and numerical analysis: once documents have been mapped into a common matrix, established methods from linear algebra, multivariate statistics, and machine learning can operate on their distributional structure.