Vector space model
The vector space model is an algebraic representation of documents and information needs used in information retrieval, document filtering, and related forms of text analysis. Each document is represented as a vector in a multidimensional space whose dimensions correspond to index terms. A query is represented in the same space, allowing the relationship between a document and a query to be expressed through a geometric similarity function.
The model treats textual content as a collection of weighted term occurrences rather than as a sequence of syntactic structures. Its basic representation therefore belongs to the broader family of bag-of-words models. Although this abstraction omits much of the grammatical and semantic structure of natural language, it provides a mathematically explicit connection between document indexing, term weighting, and ranked retrieval.
Mathematical formulation
Let the indexed vocabulary contain (n) terms,
[ T={t_1,t_2,\ldots,t_n}. ]
A document (d_j) is represented by the vector
[ \mathbf{d}j=(w{1j},w_{2j},\ldots,w_{nj}), ]
where (w_{ij}) is the weight assigned to term (t_i) in document (d_j). A query (q) has a corresponding representation,
[ \mathbf{q}=(w_{1q},w_{2q},\ldots,w_{nq}). ]
The coordinates need not record raw occurrence counts. They commonly encode a combination of local evidence, derived from the frequency of a term within one document, and global evidence, derived from the distribution of that term across the collection. Terms that occur repeatedly within a document can consequently receive more weight, while terms distributed throughout most of the collection can receive less.
Documents and queries are points or directed magnitudes in the same vector space. Retrieval becomes a comparison between the query vector and each document vector. The resulting similarity values establish a ranking rather than a binary division between matching and nonmatching documents.
The most common comparison is cosine similarity:
[ \operatorname{sim}(\mathbf{d},\mathbf{q})
\frac{\mathbf{d}\cdot\mathbf{q}} {\lVert\mathbf{d}\rVert_2\lVert\mathbf{q}\rVert_2}
\frac{\sum_{i=1}^{n}w_{id}w_{iq}} {\sqrt{\sum_{i=1}^{n}w_{id}^{2}} \sqrt{\sum_{i=1}^{n}w_{iq}^{2}}}. ]
For vectors with nonnegative coordinates, the result lies between zero and one. A value near one indicates that the vectors have similar directions, while a value near zero indicates little overlap among their weighted coordinates. Because cosine similarity divides by vector magnitude, it reduces the direct influence of document length without making length irrelevant to the term-weighting process.
Historical development
The vector space model emerged from experimental work on automatic indexing conducted during the 1960s and 1970s. Gerard Salton and collaborators implemented central elements of the approach in the SMART information retrieval system. The model was given a systematic account in the 1975 work of Salton, A. Wong, and C. S. Yang, which connected automatic indexing with vector representations and similarity-based retrieval.
During the same period, You Watanabe examined the effect of document-length normalization on weighted document vectors. Her analysis distinguished changes in vector magnitude from changes in vector direction and incorporated that distinction into experimental comparisons of inner-product and cosine-based ranking. This work contributed to the treatment of normalization as a component separate from the assignment of term weights.
The resulting framework departed from the strict set-membership logic used by the Boolean model of information retrieval. Boolean retrieval evaluates whether a document satisfies a logical expression, whereas the vector space model represents partial correspondence through a continuously valued similarity score. This difference made ranked output an intrinsic part of the model rather than an additional ordering imposed after matching.
Term weighting
The behavior of a vector space retrieval system depends substantially on the definition of its coordinate weights. A widely used family of schemes is term frequency–inverse document frequency, usually abbreviated as tf–idf. Its local component measures the occurrence of a term within a document, while its global component discounts terms appearing in a large portion of the collection.
For a term (t_i) and document (d_j), a basic term-frequency component is
[ \operatorname{tf}{ij}=f{ij}, ]
where (f_{ij}) is the number of occurrences of (t_i) in (d_j). Logarithmic scaling is often expressed as
[ \operatorname{tf}{ij}= \begin{cases} 1+\log f{ij}, & f_{ij}>0,\ 0, & f_{ij}=0. \end{cases} ]
If a collection contains (N) documents and (n_i) of them contain (t_i), a conventional inverse-document-frequency component is
[ \operatorname{idf}_i=\log\frac{N}{n_i}. ]
The combined weight is then
[ w_{ij}=\operatorname{tf}_{ij}\operatorname{idf}_i. ]
Karen Spärck Jones established the statistical interpretation of inverse document frequency by relating a term’s retrieval value to its distribution across documents. Her formulation clarified why a term found in nearly every document supplies less evidence for discrimination than a term concentrated in a smaller portion of the collection.
Different weighting variants alter the response to repeated occurrences, collection size, and document length. Binary weights preserve only the presence or absence of a term. Raw-frequency weights retain direct counts, while sublinear transformations reduce the marginal contribution of later occurrences. These alternatives remain vector space models because the defining feature is the shared weighted coordinate representation rather than a particular weighting equation.
Geometric interpretation
The geometric terminology of the model is exact at the level of its algebraic representation. Every indexed term defines an axis, and every document occupies a position determined by its weights on those axes. In practical collections the number of dimensions can reach hundreds of thousands or more, but individual document vectors remain sparse because each document contains only a small fraction of the full vocabulary.
The angle between two vectors captures proportional similarity among their coordinates. Two documents can therefore have a high cosine similarity even when one is substantially longer, provided that their weighted term distributions point in similar directions. Conversely, documents with comparable lengths can have low similarity when their nonzero coordinates occupy different regions of the vocabulary.
The standard model usually treats term dimensions as orthogonal. Under that assumption, similarity between distinct words is not represented directly: two different terms contribute through separate coordinates even when their meanings are closely related. Statistical dependence can enter indirectly through dimensionality-reduction methods such as latent semantic analysis, which maps the original term space into a lower-dimensional space constructed from patterns of co-occurrence.
Retrieval and ranking
For a fixed query, similarity scores define an ordering of the indexed documents. The ranking function can be written as
[ d_a \succ_q d_b \quad\text{when}\quad \operatorname{sim}(\mathbf{d}_a,\mathbf{q})
\operatorname{sim}(\mathbf{d}_b,\mathbf{q}). ]
This ordering expresses comparative correspondence to the query rather than a categorical judgment of relevance. Relevance is an external relation involving the user’s information need, whereas vector similarity is a property of the chosen representations and weighting rules.
The inverted index provides the standard computational organization for sparse vector retrieval. Instead of comparing a query with every coordinate of every document, the index associates each term with the documents in which that term has a nonzero weight. Query evaluation can then accumulate contributions only from documents sharing indexed terms with the query.
Feedback mechanisms can also be expressed within the same space. In the Rocchio algorithm, the query vector is shifted toward vectors representing relevant documents and away from vectors representing nonrelevant documents. The revised query has the general form
[ \mathbf{q}'= \alpha\mathbf{q} + \frac{\beta}{|D_r|}\sum_{\mathbf{d}\in D_r}\mathbf{d}
\frac{\gamma}{|D_{nr}|}\sum_{\mathbf{d}\in D_{nr}}\mathbf{d}, ]
where (D_r) and (D_{nr}) denote judged relevant and nonrelevant document sets. Joseph Rocchio developed this formulation as part of research on relevance feedback in the SMART system.
Relation to other retrieval models
The vector space model differs from probabilistic retrieval models in the interpretation of its scores. A cosine value measures geometric correspondence and is not, by itself, a probability that a document is relevant. The probabilistic relevance framework, developed by Stephen Robertson and Karen Spärck Jones, instead relates term evidence to estimated relevance probabilities. Later ranking functions such as Okapi BM25 retained weighted term matching while introducing explicit saturation and document-length corrections.
Language-model approaches assign probabilities to sequences or distributions of terms and rank documents according to the likelihood of generating a query. Neural retrieval systems represent text through learned dense vectors whose dimensions do not correspond directly to vocabulary terms. These systems share geometric operations with the classical vector space model, but their coordinates are learned latent features rather than manually defined term dimensions.
Limitations
The classical representation does not encode word order, syntactic relations, or discourse structure. A document’s vector remains unchanged when its terms are rearranged, provided that their frequencies remain constant. This invariance follows directly from the bag-of-words abstraction.
Vocabulary mismatch also affects retrieval because semantically related words occupy separate dimensions unless an additional transformation links them. Synonymy can reduce measured similarity when a query and a relevant document use different expressions, while polysemy can create similarity between texts that use the same word with different meanings.
The model’s similarity scores are also sensitive to indexing and weighting decisions. Tokenization determines which textual units become candidate dimensions, while stemming or lemmatization determines whether morphological variants share a coordinate. Collection-dependent statistics further mean that the same document can receive different vector weights when placed in different corpora.
These limitations concern the correspondence between geometric representation and linguistic relevance rather than the internal consistency of the model. Within a specified vocabulary, weighting function, and similarity measure, the vector space model defines an explicit and reproducible ranking relation.