RAG: How Retrieval-Augmented Generation Lets Language Models Answer from Your Documents

A language model trained six months ago cannot answer questions about what happened last week. It cannot cite your internal documents, your company's policies, or a database it has never seen. RAG, short for Retrieval-Augmented Generation, solves this by giving the model a library card instead of a brain transplant. Rather than storing facts in model weights, a RAG system looks them up at query time: it retrieves relevant passages from an external corpus and places them directly in the prompt as context. The model then generates an answer grounded in what it just read, not in what it memorized during training.

The paper that named the technique

The term and the formal architecture come from 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' by Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela, submitted to arXiv on May 22, 2020 (arXiv:2005.11401) and published at NeurIPS 2020. The paper framed the problem as separating two types of model memory: parametric memory, knowledge baked into the weights through training; and non-parametric memory, a separate retrievable index that can be updated without retraining. The non-parametric store in the original paper was a dense vector index of the full English Wikipedia, roughly 21 million 100-word passages. On three open-domain question-answering benchmarks, RAG set state-of-the-art results at the time, outperforming both pure parametric models and traditional retrieve-and-extract pipelines.

How the two-stage pipeline works

A RAG system runs two stages in sequence. In the retrieval stage, the user's query is converted into a dense vector embedding by an encoder model. That vector is compared against a pre-built index of all documents in the knowledge base, also encoded as vectors, and the top-k most similar passages are returned. The comparison is typically a cosine similarity or inner product computation over many stored vectors, made fast by libraries such as FAISS, released by Meta AI and supporting approximate nearest-neighbor search at billion-vector scale. In the generation stage, the retrieved passages are concatenated with the original query and the combined text is placed in the model's context window. The language model then generates an answer conditioned on both the query and the retrieved evidence.

The Lewis et al. paper compared two variants: RAG-Sequence, which conditions on the same set of retrieved passages across the full output, and RAG-Token, which can draw on different passages at each token generation step. RAG-Sequence performed better on most benchmarks tested in the paper, and the simpler retrieve-once design is what practitioners typically build today.

Dense retrieval versus keyword search

Before dense retrieval, the standard approach was BM25, a sparse keyword-matching method that scores passages based on term frequency weighted against how rare the query terms are across the corpus. BM25 is fast and requires no neural computation, but it fails when the query and the relevant passage use different words to describe the same idea. A query about 'cardiac arrest treatment' may miss a passage that uses only 'heart attack management.' Dense Passage Retrieval (DPR), published at EMNLP 2020 (arXiv:2004.04906) by Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih at Facebook AI Research, showed that a dual-encoder trained on question-passage pairs learns semantic similarity beyond keyword overlap. DPR outperformed a strong BM25 baseline by 9 to 19 percentage points in top-20 passage retrieval accuracy across multiple open-domain question-answering datasets. Hybrid search combining BM25 term scores with dense similarity scores generally improves recall further on hard queries without hurting performance on easy ones.

How to structure the prompt around retrieved passages

How you construct the prompt around the retrieved passages matters as much as what you retrieve. Several patterns hold up across production systems.

• Place retrieved context before the user's question. The model attends to early context more reliably than to late context when generating the first tokens of the answer.
• Label each retrieved passage with a source marker such as [Document 1], [Document 2] so the model can attribute specific claims and a downstream citation step can link answers back to sources.
• Include an explicit instruction to use only the provided context: 'Answer the question using only the information in the passages below. If the answer is not in the passages, say you do not know.' This reduces hallucination more reliably than leaving the model to decide for itself.
• When retrieving multiple passages, put the highest-ranked passage closest to the question. Models show a recency bias in longer contexts, so the passage most likely to contain the answer benefits from being nearest to the query.
• Truncate each retrieved chunk to a fixed token budget of 100 to 300 tokens. Very long chunks dilute the signal and can push the actual answer far from the question in the context window.

Common failure modes and how to address them

RAG fails in predictable ways. The most common is a retrieval miss: the retriever returns passages that are topically related but do not contain the answer. The model then either hallucinates a plausible-sounding answer or hedges unhelpfully. Fixing this requires improving retrieval quality, not adjusting the generation prompt. Hybrid search combining BM25 and dense scores, together with cross-encoder reranking of the top retrieved results, both reduce this failure mode substantially. A second failure is context overflow: when retrieved passages are long and numerous, the model may lose track of the question or underweight later evidence. Limiting chunk size and retrieval count controls this. A third failure is instruction drift, where the model begins ignoring the 'use only provided context' constraint partway through a long prompt. Repeating the core instruction just before the question helps.

When RAG fits and when it does not

RAG is well matched to problems where the information the model needs is too recent for training data, too specific to appear in a general corpus, or too sensitive to embed in model weights. Customer support over product documentation, legal review of internal contracts, enterprise knowledge base search, and scientific literature question answering all fit this pattern. RAG is less appropriate when the task requires synthesizing information scattered across many documents rather than locating a single relevant passage, or when the query is ambiguous enough that retrieval surfaces mostly noise. In those cases, a long-context model loaded with relevant documents upfront, or an agent that retrieves and reasons iteratively rather than once, is usually a better fit. PromptingIndex covers prompt patterns for RAG pipelines tested across Claude, ChatGPT, and Gemini, including designs that minimize hallucination when retrieved context is partial or absent.