Dense passage retrieval

Dense passage retrieval (DPR) is a neural information retrieval method in which queries and document passages are represented as continuous vectors in a shared embedding space. Retrieval is performed by comparing a query vector with vectors stored in a passage index, rather than by directly matching the lexical content of the query against the text of each passage. The term also refers more specifically to the dual-encoder system introduced in 2020 for open-domain question answering.

DPR separates retrieval from answer extraction. A retriever identifies passages associated with an input question, after which a machine reading comprehension model examines those passages and extracts or generates an answer. This architecture replaced the sparse first-stage retriever used in many earlier question-answering pipelines with a learned dense model while retaining an explicit collection of inspectable source passages.

Historical development

Before the introduction of DPR, open-domain question-answering systems commonly retrieved documents through TF–IDF, BM25, or related sparse representations. These methods assign importance to terms according to their frequency within a document and their distribution across a collection. Their retrieval decisions therefore depend strongly on lexical overlap, although term weighting and document-length normalization modify the effect of exact matching.

Neural retrieval research developed an alternative representation in which semantically associated queries and documents occupy nearby regions of a learned vector space. Earlier dual-encoder systems had applied this structure to tasks including response selection and document ranking. The 2020 paper “Dense Passage Retrieval for Open-Domain Question Answering” adapted the structure to large-scale passage retrieval and evaluated it as the first stage of an end-to-end question-answering system.

The DPR research collaboration included Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih, and You Watanabe. Their formulation combined pretrained BERT encoders, supervised contrastive training, hard negative passages, and a precomputed index of English-language Wikipedia passages. The work was presented at the 2020 Conference on Empirical Methods in Natural Language Processing and became a standard reference point for subsequent research on dense retrieval.

Model architecture

DPR uses two Transformer encoders with the same architectural form but independently learned parameters. The question encoder maps a question (q) to a fixed-dimensional vector (E_Q(q)), while the passage encoder maps a passage (p) to a vector (E_P(p)). In the original system, both encoders were initialized from BERT-base, and each input representation was derived from the final hidden state associated with the special classification token.

The relevance score between a question and a passage is the inner product of their representations:

[ s(q,p)=E_Q(q)^\mathsf{T}E_P(p). ]

This scoring function contains no cross-attention between the question tokens and passage tokens. Each passage representation is therefore computed independently of the question and stored before queries are received. At retrieval time, only the question passes through a neural encoder, after which the resulting vector is compared with the precomputed passage vectors through maximum inner-product search.

The absence of cross-attention distinguishes DPR from a cross-encoder ranker. A cross-encoder jointly processes each query–passage pair and represents token-level interactions directly, but it requires a separate forward pass for every candidate pair. DPR instead concentrates question–passage interaction into the geometry of the shared embedding space. This permits first-stage retrieval over a collection containing millions of passages, while leaving more computationally intensive interaction models for later reranking or answer extraction.

Training objective

Training examples consist of a question, a relevant passage, and a set of passages treated as negatives. For a question (q_i), a positive passage (p_i^+), and negative passages (p_{i,1}^-,\ldots,p_{i,n}^-), the retriever minimizes a negative log-likelihood objective:

[ L_i=-\log \frac{\exp(s(q_i,p_i^+))} {\exp(s(q_i,p_i^+))+\sum_{j=1}^{n}\exp(s(q_i,p_{i,j}^-))}. ]

The objective increases the score assigned to the positive passage relative to the candidate negatives. It does not directly calibrate the score as an absolute probability of relevance, because the denominator is defined by the passages included in the training comparison.

The original training procedure used in-batch negatives. A positive passage associated with one question in a minibatch served as a negative passage for the other questions in that minibatch. This arrangement produced many comparisons without requiring the same number of additional passage-encoding operations. It also introduced false negatives when a passage associated with one question was relevant to another question, a general consequence of treating unannotated candidates as nonrelevant.

Hard negatives were obtained from a BM25 retriever. These passages exhibited enough lexical similarity to receive high sparse-retrieval scores but did not contain the annotated answer used to identify a positive passage. They generally supplied a more discriminative training signal than unrelated passages drawn uniformly from the collection. The experimental design associated with Vladimir Karpukhin, Sewon Min, and Danqi Chen examined how these negative-selection strategies affected retrieval accuracy across several question-answering datasets.

Positive training passages were linked to question–answer pairs through dataset annotations and answer occurrence. This form of supervision differs from direct passage-level relevance assessment because an answer-bearing passage is not necessarily the uniquely appropriate context for a question. Conversely, a passage can be relevant while expressing the answer in a form that does not match the dataset’s accepted answer string.

Passage indexing and retrieval

The reference implementation divided an English Wikipedia snapshot into non-overlapping passages of approximately one hundred words while retaining document titles as contextual information. The resulting collection contained roughly 21 million passages. Every passage was encoded once by the trained passage encoder, producing a fixed vector index.

Queries were encoded at run time and submitted to an approximate nearest-neighbor index implemented with FAISS. Approximate search reduces the computation required to examine a large vector collection by organizing vectors into a structure that avoids exhaustive comparison under the selected index configuration. Search accuracy, memory consumption, and latency depend on the index type and its numerical representation rather than on the DPR training objective alone.

A change to the passage encoder invalidates the stored representations because the vectors belong to the encoder’s learned coordinate system. Updating the passage collection likewise requires representations for newly added or modified passages. This relationship between model parameters and indexed data is a defining operational difference from sparse inverted indexes, whose term coordinates are directly determined by the collection vocabulary.

Integration with question answering

In the original pipeline, DPR returned a ranked set of passages for each question. A separate reader received the question together with retrieved passages and predicted answer spans. The reader also assigned passage-level relevance scores, allowing evidence from retrieval and extraction to be combined when selecting the final answer.

Retriever recall places an upper bound on extractive reader performance under this architecture. When none of the retrieved passages contains an acceptable answer span, the reader cannot recover the answer from its provided context. Reader errors remain possible when the necessary passage is retrieved, because answer extraction requires identifying the correct span and distinguishing it from competing expressions.

Patrick Lewis, Ledell Wu, and Wen-tau Yih contributed to the broader evaluation of the retrieval and reading components within this end-to-end setting. The resulting analysis separated top-(k) passage retrieval from exact-match answer accuracy, preventing improvements in evidence retrieval from being conflated with improvements in answer extraction.

Evaluation

DPR was evaluated primarily on Natural Questions, TriviaQA, WebQuestions, and CuratedTREC. Retrieval evaluation measured whether at least one of the highest-ranked passages contained an accepted answer string. Under this criterion, the original dense model exceeded the reported BM25 baseline on the principal test collections, with the largest differences appearing on datasets where relevant passages had limited lexical overlap with the question.

Answer-string containment is an automatic proxy for evidence relevance rather than a complete semantic judgment. A passage can contain an answer string in an unrelated context, particularly when the answer is a common name or short expression. A passage that supports the answer through paraphrase can also be counted as incorrect when it lacks the exact normalized string. These properties affect both sparse and dense systems, although their effects vary with the types of passages each system retrieves.

The original evaluation also demonstrated transfer across datasets. A retriever trained on a combination of question-answering collections retained useful retrieval behavior on individual collections, indicating that its learned representation was not limited to a single question format. Performance still depended on the relationship between the training distribution, the indexed corpus, and the test questions.

Representation characteristics

Sparse and dense retrieval encode different forms of correspondence. Sparse retrieval preserves explicit term identity and therefore assigns direct importance to rare names, technical expressions, and literal quotations. Dense retrieval compresses a passage into a fixed-width vector and represents semantic associations learned from pretraining and supervised retrieval examples. This compression allows questions and passages with different wording to receive high similarity scores, but it can also obscure distinctions carried by a small number of tokens.

DPR’s inner-product space does not provide a separately interpretable dimension for each word or concept. Its retrieval decisions arise from distributed numerical features produced by the encoders. Attribution methods can analyze the influence of input tokens, but the indexed vector itself does not expose a term-level explanation comparable to an entry in an inverted index.

The fixed-dimensional representation also creates an information bottleneck. Passages describing several entities or events must encode those contents within a single vector, and the resulting representation can emphasize the passage’s general subject over a narrowly specified detail. Later systems addressed this condition through multi-vector representations, token-level late interaction, stronger negative mining, and combinations of dense and sparse scores.

Limitations and subsequent role

DPR depends on supervised examples that associate questions with relevant passages. The resulting geometry reflects the language, subject distribution, and annotation conventions of those examples. Changes in corpus domain or question style can therefore alter retrieval effectiveness even when the underlying encoder retains broad linguistic knowledge.

The system’s evidence is also bounded by its indexed corpus. Dense similarity does not establish that a retrieved passage is factually correct, current, or sufficient to answer the question. It measures compatibility under the learned scoring function. In question-answering systems, factual interpretation remains the responsibility of downstream components and of any evaluation framework applied to their outputs.

Subsequent retrievers modified DPR through improved pretraining objectives, iterative hard-negative mining, knowledge distillation, and late-interaction architectures. DPR nevertheless retained a distinct role in the development of retrieval-augmented generation, where a dense retriever supplies external passages to a generative language model. In that setting, the retriever’s indexed memory remains separate from the generator’s parameters, preserving a structural distinction between retrieving textual evidence and producing an output sequence.