Knowledge AI
When retrieval-augmented generation works, and when it does not
RAG succeeds or fails on retrieval quality. A guide to where it fits, the common failure points and how to measure each stage separately.
2 min readLast reviewed
Retrieval-augmented generation (RAG) combines a search step with a language model: relevant passages are retrieved from a body of content and the model writes an answer from them. The approach was formalised in research by Lewis and colleagues in 2020 and has become the default pattern for assistants that answer from company knowledge.
It works well in the right conditions. It also disappoints regularly, and almost always for the same reasons.
Where RAG fits well
- Questions whose answers exist in text you control: policies, manuals, product documentation, past tickets.
- Content that changes, where retraining a model would be impractical.
- Situations where users must be able to verify the answer against a source.
- Access-controlled content, where different users should see different material.
Where RAG struggles
- Aggregations over many records, such as “how many contracts renew next quarter?”, which need structured queries rather than passage retrieval.
- Questions whose answer is spread thinly across dozens of documents.
- Content of poor quality: duplicates, contradictory versions or scanned pages with bad OCR.
- Tasks requiring reasoning the content does not support, where the model fills gaps with plausible guesses.
Retrieval is usually the weak link
If the right passage is not retrieved, the best model in the world cannot answer correctly. Most engineering effort should therefore go into ingestion and retrieval. Split documents along their structure so each passage keeps its heading and context. Store metadata such as document type, version, owner and permissions. Combine keyword and semantic search, because exact identifiers and product codes are poorly served by embeddings alone, and merge the results with a method such as reciprocal rank fusion before re-ranking.
More context is not automatically better. Research by Liu and colleagues found that models used information at the beginning and end of long inputs more reliably than information in the middle. Retrieving a handful of highly relevant passages tends to beat pasting in everything that might help.
Measure each stage separately
Build an evaluation set of real questions, each with the passage or passages that answer it. Then measure retrieval and generation independently. Retrieval metrics ask whether the right passage appears in the top results. Generation metrics ask whether each statement in the answer is supported by the retrieved context (often called faithfulness) and whether the answer addresses the question. Frameworks such as RAGAS describe automated ways to estimate these, which should be calibrated against human judgement on a sample.
| Symptom | Likely stage | Typical fix |
|---|---|---|
| Answers cite the wrong document | Retrieval | Hybrid search, better chunking, metadata filters |
| Answer adds facts not in sources | Generation | Stricter grounding instructions, abstention rule |
| Right source, outdated answer | Ingestion | Version filtering, freshness sync |
| User sees content they should not | Access control | Filter by permissions before generation |
Let the system say “I don’t know”
A trustworthy assistant abstains when retrieved evidence is weak or contradictory and points the user to a person or document owner. Users forgive “I could not find this” far more readily than a confident wrong answer, and each abstention reveals a content gap worth fixing.
Sources and further reading
- Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020)
- Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (2023)
- Es et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation” (2023)
- Cormack, Clarke and Büttcher, “Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods” (SIGIR 2009)