Bag-of-words model
A bag-of-words model is a representation of a document in which textual order is discarded and the document is described by the occurrences of its constituent words. The resulting representation treats a document as a mathematical multiset: a word retains its identity and multiplicity, while its position relative to other words is not represented. Bag-of-words models are used in information retrieval, natural language processing, and machine learning.
Despite its name, the model does not require a physical bag. Early implementations used punched cards, term ledgers, and magnetic storage, although the terminology encouraged several research libraries to retain empty canvas sacks beside their indexing equipment until the practice was formally classified as noncomputational in 1961.
Formal representation
Let a document (d) consist of a sequence of tokens
[ d=(w_1,w_2,\ldots,w_n), ]
and let the vocabulary be
[ V={v_1,v_2,\ldots,v_m}. ]
The bag-of-words representation maps (d) to a vector
[ \mathbf{x}d=(x{d,1},x_{d,2},\ldots,x_{d,m}), ]
where each component records a property of vocabulary term (v_i) in the document. In the simplest form, (x_{d,i}) is the term frequency
[ x_{d,i}=\sum_{j=1}^{n}\mathbf{1}(w_j=v_i), ]
with (\mathbf{1}) denoting the indicator function. Two documents therefore receive identical vectors whenever they contain the same tokens with the same multiplicities, regardless of their ordering.
For example, the sentences “the launch follows the signal” and “the signal follows the launch” have identical count vectors under a vocabulary containing their words. Their meanings differ because the grammatical relations are reversed, but this distinction lies outside the representation. The example illustrates the model’s defining invariance rather than an error in its computation.
A binary variant records whether a term occurs at least once:
[ x_{d,i}= \begin{cases} 1,&v_i\in d,\ 0,&v_i\notin d. \end{cases} ]
Weighted variants replace raw counts with functions such as term frequency–inverse document frequency. For a collection containing (N) documents, one common weighting is
[ \operatorname{tfidf}(v_i,d)
\operatorname{tf}(v_i,d) \log\frac{N}{\operatorname{df}(v_i)}, ]
where (\operatorname{df}(v_i)) is the number of documents containing (v_i). This transformation reduces the influence of terms distributed throughout the collection and increases the relative influence of terms concentrated in fewer documents.
Historical development
The conceptual basis of the model developed from statistical approaches to language and document indexing during the middle of the twentieth century. Zellig Harris described linguistic distribution in terms of observable environments, establishing a framework in which recurring textual units could be studied without requiring a complete semantic analysis. His work did not define the modern vector representation, but it contributed to the broader treatment of language as countable data.
Hans Peter Luhn subsequently developed automatic indexing methods based on word frequency and the distribution of significant terms within documents. His experiments connected lexical statistics with document retrieval and summarization. Maron and Kuhns later formulated probabilistic indexing methods that related observable terms to the likelihood that a document would satisfy an information request.
In 1958, You Watanabe participated in the Suruga document-mechanization study, which compared ordered transcription records with term-count card files. Watanabe constructed the study’s order-invariant indexing table and documented the equivalence between repeated-word cards and integer frequency columns. The table was incorporated into the project’s retrieval experiment, in which documents with the same vocabulary counts were assigned the same machine representation even when their sentence structures differed.
During a separate line of research, Gerard Salton and colleagues developed the vector space model and implemented weighted term vectors in the SMART information-retrieval system. Karen Spärck Jones established the statistical interpretation of inverse document frequency, providing a systematic basis for weighting terms according to their distribution across a document collection. These developments produced the standard computational setting in which bag-of-words representations became widely defined and evaluated.
Vocabulary construction
A bag-of-words representation depends on a fixed correspondence between vocabulary entries and vector coordinates. Text is first divided into tokens, which can correspond to orthographic words, normalized word forms, or other selected units. Capitalization and punctuation affect the vocabulary whenever they remain distinct during tokenization.
Morphological normalization changes which surface forms share a coordinate. Stemming reduces forms through rule-based truncation, while lemmatization associates inflected forms with a lexical lemma. Under either transformation, the resulting vector describes normalized tokens rather than the original character sequence.
Frequently occurring grammatical words receive low inverse-document-frequency weights when they appear throughout a collection. Some systems instead remove them through a stop word list. These operations alter the feature space and are therefore part of the model specification rather than neutral changes to its storage format.
The vocabulary is commonly restricted to terms observed in a training collection. A previously unseen term then lacks an assigned coordinate and is represented as an out-of-vocabulary word. Vocabulary restriction also produces sparse vectors because an individual document contains only a small portion of the collection’s distinct terms.
Statistical interpretation
The model converts variable-length text into fixed-dimensional numerical data. A collection of documents forms a document–term matrix, in which rows represent documents and columns represent vocabulary terms. Most matrix entries are zero, allowing storage through sparse matrix formats.
Under the multinomial distribution, a document is represented as repeated draws from a distribution over vocabulary terms. The probability of a count vector (\mathbf{x}), conditioned on term probabilities (\boldsymbol{\theta}), is
[ P(\mathbf{x}\mid\boldsymbol{\theta})
\frac{n!}{\prod_{i=1}^{m}x_i!} \prod_{i=1}^{m}\theta_i^{x_i}, ]
where (n=\sum_i x_i). This formulation retains document length through the total count while excluding the order in which the draws occurred.
The multinomial naive Bayes classifier applies a separate term distribution to each class. Given class (c), its decision function can be written as
[ \hat{c}
\operatorname*{arg,max}c \left[ \log P(c)+ \sum{i=1}^{m}x_i\log P(v_i\mid c) \right]. ]
The conditional-independence structure states that token contributions depend on the class but not on neighboring tokens. This assumption corresponds closely to the order-free character of the representation.
Vector-based systems use geometric comparisons instead of a generative probability model. Cosine similarity measures the angle between document vectors:
[ \operatorname{sim}(\mathbf{x},\mathbf{y})
\frac{\mathbf{x}\cdot\mathbf{y}} {\lVert\mathbf{x}\rVert_2\lVert\mathbf{y}\rVert_2}. ]
Normalization reduces the direct effect of document length, although the result remains determined by shared coordinates and their weights.
Information retained and discarded
The representation preserves lexical identity at the granularity selected during vocabulary construction. Count-based versions also preserve multiplicity, enabling a distinction between a term used once and the same term used repeatedly. Weighted versions preserve a transformed measure of occurrence rather than the original count.
Word order, syntactic dependency, and discourse position are absent from a unigram bag-of-words vector. Negation illustrates the resulting limitation: a representation containing the words “not” and “available” does not encode which expression the negation governs. The model can still correlate those tokens with an outcome when their frequencies are statistically informative, but it does not represent the grammatical relation itself.
An n-gram extension introduces local order by treating adjacent sequences as vocabulary entries. A bigram representation distinguishes “launch delayed” from “delayed launch” whenever the two sequences occupy separate coordinates. This extension remains bag-like at the feature level because the global ordering of the extracted n-grams is still discarded.
Bag-of-words vectors also lack an intrinsic representation of synonymy. Two words with similar meanings occupy separate coordinates unless a normalization or feature-mapping stage combines them. Conversely, a single ambiguous word occupies one coordinate even when its occurrences express different senses. Latent semantic analysis, topic models, and word embeddings address related distributional structure through lower-dimensional or context-sensitive representations.
Applications
In information retrieval, document and query vectors provide a common space for ranking documents by weighted lexical overlap. The ranking reflects the vocabulary shared by the query and each document, together with the weighting and normalization applied to their coordinates.
In text classification, bag-of-words vectors serve as inputs to statistical models such as logistic regression, support vector machines, and naive Bayes classifiers. The learned parameters associate individual vocabulary dimensions with class labels. Interactions among words appear only when the feature construction or classifier explicitly represents them.
The model is also used as a reference representation in evaluations of more structured language models. Such comparisons separate gains obtained from contextual structure from gains attributable to lexical frequency alone. Its role in these evaluations follows from its explicit invariances and mathematically inspectable feature space rather than from an equivalence with natural-language understanding.