What is retrieval-augmented generation (RAG)?
Retrieval-augmented generation (RAG) is a technique where an application first searches a collection of documents for passages relevant to a question, then gives those passages to a language model along with the question, so the model answers from that material. It lets a model use private, recent or specialised information it was never trained on, and cite its sources.
RAG is open-book mode for a language model. Instead of hoping the model remembers the answer, you find the relevant pages first and put them in front of it.
How does RAG work?
A RAG system has two phases: preparing the documents once, and answering each question.
- 1SplitBreak documents into passages, a step called chunking.
- 2IndexTurn each chunk into an embedding and store it, usually in a vector database, often alongside a keyword index.
- 3RetrieveWhen a question arrives, search the index for the most relevant chunks.
- 4Rerank (optional)Reorder the candidates with a more precise model and keep the best few.
- 5GenerateSend the question plus the selected chunks to the LLM, with instructions to answer only from them and cite which chunk supports each claim.
The name comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, which combined a retriever over Wikipedia with a text generator. Today the term covers any system that retrieves information at question time and adds it to a model's context.
Why use RAG?
RAG solves the gaps in what a model knows without retraining it.
| Problem | How RAG helps |
|---|---|
| Model does not know your private data | Retrieves from your own documents, tickets or code |
| Model's knowledge is out of date | The index can be updated minutes after a document changes |
| Model hallucinates specifics | The answer is grounded in real passages |
| Users need to verify answers | Each claim can cite the chunk it came from |
| Access control | Retrieval can filter to documents the user is allowed to see |
Should I use RAG or fine-tuning?
Use RAG for knowledge and fine-tuning for behaviour. RAG is the right choice when answers depend on facts that change, must be cited, or differ by user. Fine-tuning is for changing how the model writes or performs a narrow task. Many production systems use both. If all your documents fit comfortably in the context window, you may not need retrieval at all: just include them, ideally with prompt caching.
Why do RAG systems give bad answers?
Most RAG failures are retrieval failures: the right passage never reaches the model. Diagnose which stage failed before changing the prompt.
| Symptom | Likely cause | Fix |
|---|---|---|
| Answer says the information is not available, but it is | The right chunk was not retrieved | Better chunking, hybrid search, query rewriting |
| Answer mixes up two products or versions | Chunks lack context, such as which product they describe | Add titles and context to each chunk before embedding |
| Answer is correct but misses part of the question | Too few chunks, or key chunk ranked low | Retrieve more candidates, then rerank |
| Answer includes claims not in the sources | The model filled gaps from its own knowledge | Instruct it to use only the sources and to say when they do not cover the question |
| Exact codes or names are not found | Pure vector search misses exact matches | Add keyword search (BM25) |
Anthropic's 2024 work on contextual retrieval shows how much retrieval quality matters: prepending a short, generated description of each chunk's context before embedding, combined with keyword search, reduced failed retrievals by 49% in its tests, and by 67% with reranking added.
How do you evaluate a RAG system?
Evaluate retrieval and generation separately. For retrieval, build a set of real questions with the passages that answer them, and measure how often the right passage appears in the top results. For generation, check answers for correctness, for faithfulness to the retrieved sources, and for citing them properly, using human review at first and an LLM judge once you have calibrated one.
- Do: Start by reading the retrieved chunks for 20 real questions, before tuning anything
- Do: Store the source, title and date with every chunk
- Do: Ask the model to quote or cite the passages it used
- Avoid: Tune the prompt when the real problem is that the right passage was never retrieved
- Avoid: Retrieve 50 chunks "to be safe"; irrelevant passages confuse the answer and cost tokens
- Avoid: Forget to re-index when documents change or are deleted
Frequently asked questions
What is RAG in simple terms?
RAG means looking up relevant information first and giving it to the AI along with the question, so the AI answers from real sources rather than from memory alone.
Does RAG eliminate hallucinations?
No, but it reduces them a lot. The model can still misread or go beyond the retrieved passages, so ask for citations and check that each cited passage supports the claim.
Do I need a vector database for RAG?
Not always. Small collections can be searched in memory, and many existing databases, such as Postgres with pgvector, support vector search. A dedicated vector database helps at larger scale.
Is RAG still needed with million-token context windows?
Often yes. Retrieval is cheaper and faster than sending huge contexts on every request, scales beyond any window, and supports access control. For small, fixed document sets, putting everything in context can be simpler.
What is agentic RAG?
Agentic RAG lets an agent decide when and how to search, rewrite queries, run several searches and judge whether it has enough information, instead of running one fixed retrieval step before answering.
Last checked for accuracy on . Written by the solidcoder team.