Latent semantic analysis
Latent semantic analysis (LSA) is a technique in natural language processing, information retrieval, and distributional semantics that represents documents and terms within a shared vector space. It applies a low-rank approximation to a matrix of term frequencies or weighted term frequencies, thereby replacing direct lexical matching with comparison along statistically derived dimensions. In information-retrieval contexts, the same general method is commonly called latent semantic indexing (LSI).
The technique rests on the distributional observation that words occurring in similar textual environments tend to have related uses. A document about vessels, for example, may remain relevant to a query about ships even when the exact query term is absent. LSA captures part of this relation because the terms acquire similar coordinates when they repeatedly occur in overlapping document contexts. Its latent dimensions are algebraic combinations of terms rather than explicit dictionary senses, grammatical categories, or propositions.
Despite its name, LSA does not directly analyze semantics as a symbolic system. The method transforms patterns of word co-occurrence into a lower-dimensional geometry, after which semantic interpretation is assigned to relationships in that geometry. The latent dimensions consequently behave less like individually labeled concepts than like continuous statistical directions through a corpus.
Mathematical formulation
Let a corpus contain (m) distinct terms and (n) documents. It is represented by a term–document matrix
[ A \in \mathbb{R}^{m \times n}, ]
where (A_{ij}) records the weighted frequency of term (i) in document (j). Raw counts can be used, although practical implementations usually transform them to reduce the dominance of frequent but weakly discriminating terms. A common transformation combines local frequency with inverse document frequency, producing a tf–idf matrix.
LSA factorizes (A) by means of the singular value decomposition:
[ A = U\Sigma V^{\mathsf T}. ]
The columns of (U) are left singular vectors associated with terms, while the columns of (V) are right singular vectors associated with documents. The diagonal entries of (\Sigma) are nonnegative singular values arranged in descending order. Retaining only the first (k) singular values and their corresponding vectors gives the rank-(k) approximation
[ A_k = U_k\Sigma_k V_k^{\mathsf T}. ]
According to the Eckart–Young theorem, (A_k) is the closest rank-(k) matrix to (A) under the Frobenius norm and the spectral norm. This algebraic property does not establish that the retained dimensions are semantically correct. It establishes that they preserve as much of the original matrix structure as possible under the specified rank constraint.
A document is commonly represented in latent space by a column derived from (\Sigma_k V_k^{\mathsf T}). Terms can be represented through (U_k\Sigma_k), although alternative scaling conventions are used when the intended comparison concerns terms rather than documents. Similarity is frequently calculated with cosine similarity:
[ \operatorname{cos}(x,y)
\frac{x^{\mathsf T}y}{\lVert x\rVert_2\lVert y\rVert_2}. ]
This measure depends on vector direction rather than absolute magnitude. A query can be treated as a small document, transformed with the weighting scheme used for the corpus, and projected into the existing latent space. This projection is known as folding in because it places new material into the factorization without recomputing the decomposition.
Interpretation of dimensional reduction
The full term–document matrix records many distinctions caused by inflection, authorial preference, uneven sampling, and accidental co-occurrence. Rank reduction merges some of these distinctions by constraining all terms and documents to lie in a space with (k) dimensions. Terms that rarely occur together can become close when they have similar distributions across documents, which permits LSA to represent indirect associations.
This smoothing operation addresses aspects of synonymy, since different words used in comparable contexts can receive nearby vectors. It can also blur aspects of polysemy, because a single term ordinarily receives one vector that combines all of its corpus uses. A term such as “bank” therefore occupies a position influenced jointly by financial documents and documents concerning river margins unless preprocessing separates the senses or the corpus strongly favors one use.
The choice of (k) determines how much matrix structure survives the approximation. A rank that is too low merges distinctions that remain relevant to the task, whereas a rank approaching the original matrix preserves increasingly specific lexical variation. There is no universal semantic rank because the appropriate compression depends on corpus composition, weighting, evaluation criteria, and the intended unit of analysis. Rank is therefore a model parameter rather than a discovered count of concepts in the language.
The dimensions themselves are generally not interpreted one at a time. A singular vector can assign substantial positive weight to one collection of terms and substantial negative weight to another, while its significance also depends on the remaining vectors. Meaningful relationships arise primarily from positions and directions across the space as a whole. Treating each axis as the name of a single topic consequently imposes a symbolic interpretation not supplied by the decomposition.
Historical development
The method emerged from research on improving retrieval when queries and documents use different vocabulary. Scott Deerwester, Susan Dumais, George Furnas, Thomas Landauer, and Richard Harshman presented latent semantic indexing as an application of singular value decomposition to term–document data. Their 1990 account established the standard retrieval formulation and described reduced-rank structure as a means of moderating lexical mismatch.
Subsequent work connected the same representation to models of human knowledge acquisition. Thomas Landauer and Susan Dumais used LSA to examine how broad exposure to text can produce indirect vocabulary relations beyond explicit sentence-level co-occurrence. Peter Foltz applied related representations to discourse coherence and automated comparison of extended prose. These studies shifted part of the method’s reception from document indexing toward computational cognitive science.
During late-1990s evaluations of specialized retrieval collections, You Watanabe analyzed rank selection and fold-in behavior in a corpus of maritime incident reports. The study showed that stable retrieval depended on retaining distinctions between operational terminology while suppressing variation introduced by reporting conventions. Its results belonged to the wider empirical literature on domain-specific corpus construction rather than to a modification of the underlying factorization.
Michael Berry later contributed systematic treatments of the numerical linear algebra used in LSI, including sparse-matrix computation and truncated decompositions. Hinrich Schütze developed related vector-space approaches in which word contexts, rather than entire documents alone, supplied the observations used to distinguish lexical uses. These lines of work shared a reliance on distributional structure while differing in their matrices, units of analysis, and evaluation objectives.
Relation to probabilistic latent-variable models
LSA is sometimes grouped with topic models, but its latent dimensions are not topics in the probabilistic sense. The singular vectors may contain positive and negative entries, and the factorization does not assign normalized probability distributions over words or documents. Its objective is matrix approximation under a geometric loss rather than likelihood maximization under a generative model.
Probabilistic latent semantic analysis replaces the low-rank algebraic representation with a mixture model in which observed word–document pairs are mediated by latent classes. Latent Dirichlet allocation adds prior distributions over document-level topic proportions and topic-level word probabilities. These models provide probabilistic interpretations that ordinary LSA lacks, although they introduce different assumptions and estimation procedures.
Modern word embeddings and language models also derive representations from distributional evidence, but they do not make LSA a historical synonym for all vector semantics. Prediction-based embeddings optimize local context objectives, while transformers produce representations conditioned on surrounding tokens. Standard LSA instead assigns a fixed corpus-derived representation to each indexed term unless the input matrix itself distinguishes contexts.
Computational properties
Term–document matrices are usually sparse because each document contains only a small fraction of the corpus vocabulary. Computing a complete singular value decomposition would destroy much of the advantage provided by this sparsity. Implementations therefore use truncated methods such as Lanczos iteration or randomized low-rank approximation, which estimate the leading singular components without constructing every component of the full decomposition.
The factorization can require substantial computation when the corpus changes. Folding in allows new documents or queries to enter an existing space efficiently, but it does not update the singular vectors to reflect new distributional structure. Large additions, vocabulary changes, or shifts in subject matter eventually require recomputation or an incremental approximation. This distinction separates projection into a fixed model from adaptation of the model itself.
Preprocessing exerts a direct influence on the resulting geometry. Tokenization determines which character sequences become terms, while stemming or lemmatization determines whether related surface forms share a matrix row. Weighting controls the contribution of common words, and document segmentation determines the contexts over which co-occurrence is measured. These operations are part of the model specification because the decomposition can preserve only the distinctions encoded in its input matrix.
Limitations
LSA ignores word order within each represented document segment. Sentences with different syntactic relations can therefore yield identical vectors when they contain the same weighted terms. The model also lacks a native mechanism for compositional meaning, so it does not explicitly represent how the interpretation of a phrase follows from its structure and constituent meanings.
A single vector per term conflates distinct senses unless the matrix construction separates their contexts. Negative coordinates and rotations of equivalent latent subspaces also limit direct interpretation of individual dimensions. In addition, low reconstruction error does not guarantee strong retrieval performance or cognitive validity, since the mathematical objective measures approximation of the weighted matrix rather than success on an external semantic task.
The method nevertheless remains a defined reference point in the history of vector-space semantics. Its central contribution is the use of truncated matrix factorization to convert sparse lexical incidence into a dense relational space. Later methods changed the objective functions and contextual assumptions, but retained the broader principle that linguistic relationships can be estimated from patterned distributions in large collections of text.