Most RAG Hallucinations Are Retrieval Failures: How the Retrieval Brick Decides What the Model Can Invent
a confident wrong answer, the instinct is to reach for the generation knobs: a stricter prompt, a bigger model, a temperature of zero. That treats a retrieval failure as a generation failure. The model was handed the wrong pages and did its job on them. Retrieval decides what the model can invent, because it decides what the model can see.
This is a manifesto piece in Enterprise Document Intelligence (the series philosophy). It zooms in on brick 3, retrieval, with one claim: most of what teams log as LLM hallucination in RAG is not the model inventing facts. It is the model answering, faithfully, from context retrieval should never have put in front of it.
📓 The naive pipeline and the keyword fix in Sections 1 and 4 both run in the shipped notebook: parse the NIST framework yourself, print where the backup control ranks under cosine, then watch it jump to rank 1 under a plain keyword count. Repo → doc-intel/notebooks-vol1.
The whole argument fits in one picture. Take one question and one generation model, hold both fixed, and change only the retrieval in front of them. The context is the answer.
1. A measurement: the answer ranked last of 55
We ran the textbook naive pipeline on the NIST Cybersecurity Framework v1.1 (NIST CSWP 04162018, US Government work, public domain, see the NIST copyright statement), a real 55-page enterprise standard: embed every page with a sentence-transformer, embed the question, rank pages by cosine similarity, keep the top-k. The question was “what backup practices keep data available after a ransomware attack?”, and the framework answers it plainly in subcategory PR.IP-4: “Backups of information are conducted, maintained, and tested.” The entire pipeline is a dozen lines, and it runs on a laptop with no API key.
import numpy as np
from sentence_transformers import SentenceTransformer
from docintel.parsing.pdf import parse_pdf
model = SentenceTransformer("all-MiniLM-L6-v2")
lines = parse_pdf("nist_csf.pdf")["line_df"]
pages = lines.groupby("page_num")["text"].apply(" ".join)
question = "what backup practices keep data available after a ransomware attack?"
emb = model.encode([question, *pages], normalize_embeddings=True)
scores = emb[1:] @ emb[0] # cosine of every page against the question
order = np.argsort(-scores) # most similar page first
# the answer lives in subcategory PR.IP-4, on page 41
rank = list(pages.index[order]).index(41) + 1
print(rank) # 55 (out of 55 pages)That last line prints 55. The answer is not merely outside the top-k window; it is the single least similar page in the whole document, with the lowest cosine score of all.
Here is the part that should sting. The question contains the word backup, and exactly one page in the entire framework also contains it: page 41, the answer. Cosine ranked that page dead last anyway. The one word that mattered was averaged away by the rest of the question.
The page that ranked first was the Data Security category (PR.DS), which is about protecting data and is the wrong control; the rest of the top five were framework prose. Feed generation those five and it never sees the backup subcategory. It is handed data-security policy and asked how to survive ransomware, so it answers from what it was given, or fills the gap from its training, and that confident, source-free answer is what a reviewer later logs as a hallucination. The model did not invent from nothing. It answered the wrong pages.
2. The three retrieval conditions that cause hallucination
Hallucination in a grounded pipeline is almost always one of three retrieval conditions, and none of them is the model’s fault. They are worth separating because they fail differently and they leave different traces in a log. Only the first is the “invents from nothing” case people picture. The other two hand the model real, on-topic text and let it answer wrongly from it, which is why they survive review.
2.1 The answer was not retrieved
The relevant control sat at rank 55 and the top-k window ended at five. The model is asked a question whose answer is nowhere in its context, so it fills the gap from its parameters. Handed the Data Security pages and the framework’s front matter and asked how to survive ransomware, it returns something like “maintain strong access controls, encrypt data at rest, and segment the network”: fluent, security-flavoured, and supported by nothing it was given. This is the classic confident invention, and it is a recall failure upstream. It is also the only one of the three where the answer is provably absent, so it is the easiest to catch: measure the rank, see it fall outside the window, and you have the diagnosis before you read a single generated word.
2.2 The wrong passage was retrieved
A passage that sounds right but says something else wins the ranking. On this run the Data Security category (PR.DS) ranked first: PR.DS-1 data-at-rest is protected and PR.DS-2 data-in-transit is protected read, to cosine, as the answer to keep data available, because both are about protecting data. They are real controls, and they are the wrong ones. Neither is the backup subcategory.
This is the failure that survives review. The model reads PR.DS, answers from it, and often cites it, so the answer arrives looking grounded, with a valid NIST code attached. A reviewer skims the citation, sees a real subcategory, and signs off, when the code is simply aimed at the wrong control.
And it is not a resolution problem a larger embedding model fixes. A newer, bigger embedding does not know that available points at backups rather than at data-at-rest protection. On enterprise text it reorders these near-miss controls no better, and a larger model can rank them worse than a smaller one.
2.3 The answer was buried in distractors
The right subcategory is in the context, but so are a dozen near-miss controls from the same dense table. NIST prints its Framework Core as a grid, and the answer, PR.IP-4 backups of information are conducted, maintained, and tested, sits in the Information Protection block right next to PR.IP-9 response and recovery plans are in place and PR.IP-10 response and recovery plans are tested. Every one of them is about surviving an incident; only one is about backups, and they share most of their words.
Hand generation that whole page and it has to pick the backup line out of a stack of recovery-plan lines. It anchors on recovery plan instead of backups, or blends the two into a generic “keep a tested recovery plan” answer that is not what the question asked.
More retrieved context is not more grounding. The instinct to retrieve more to be safe, top-ten pages instead of the one right subcategory, makes this worse rather than better: past a point the extra pages are just noise the model has to rank in its head. The same pattern holds on any enterprise corpus where the answer is a short, precisely worded span inside a long structured document, whether that document is a security framework, an insurance policy, or a supply contract.
In all three the generation brick is doing exactly what you asked: produce an answer from the provided context. The lever that controls the outcome is upstream. And the three are not equally visible in a log. The first announces itself in the rank; the second and third arrive wearing a citation, which is why they pass review and get logged as the model’s fault instead of retrieval’s.
3. Why cosine invites all three
Top-k by cosine ranks passages by surface meaning, and surface meaning is a poor proxy for where the answer lives. The question what keeps data available after a ransomware attack? reads, to an embedding model, as incident and recovery, so cosine pulls the data-security and recovery pages up. The control that actually answers it, PR.IP-4, is phrased as backups of information are conducted, maintained, and tested. It shares exactly one word with the question, backup, and that word is outnumbered: ransomware, attack, available, and keep all drag the vector toward incident and recovery, so the one relevant word is averaged away. On top of that the control is a single terse line inside a table of dozens of subcategories, so even its own page embedding is diluted by everything printed around it. The right answer is, to cosine, the least similar page in the document.
This is not a tuning problem a bigger embedding model fixes. It is what pure similarity does on enterprise documents, where the answer is a small span whose words differ from the question’s. The retrieval brick’s own manifesto, the untaught lessons of RAG retrieval, makes the general case; hallucination is its most expensive symptom.
4. How retrieval stops it
If hallucination is a retrieval failure, it has a retrieval fix, and it is not a better embedding. It is two disciplines.
4.1 Anchor on the right span, not the nearest vector
The anchor is the point where retrieval hands off to generation, and a wrong anchor is where the model starts making things up. Find it with signals that survive vocabulary mismatch: keyword matching against an expert-validated dictionary (a security analyst maps ransomware resilience to backups, recovery, PR.IP-4, the words the framework actually uses), the document’s own structure (the framework is a tree of Functions, Categories, and Subcategories, so Protect > Information Protection Processes is an addressable node), and embeddings as one parallel vote rather than the whole decision. That is the subject of Article 7B (anchor detection): three parallel detectors, keyword, structure, and embedding, find the anchor a wrong cosine misses. When several candidates survive, Article 7C (the LLM arbiter) ranks them in a single call, each with a reason for the pick.
The simplest of those signals already settles this case, on the same run. The word the question hangs on is backup, so count it per page:
# The expert word the question hangs on is "backup".
# Count it per page and rank; no embeddings, no API.
hits = (lines[lines["text"].str.contains(r"\bbackup", case=False)]
.groupby("page_num").size()
.sort_values(ascending=False))
print(hits.index[0]) # 41 -> PR.IP-4, the answer, ranked first
print(len(hits)) # 1 -> it is the only page in the document with the wordThe word appears on exactly one page of the 55, page 41, the answer. A keyword match ranks it first; the embedding ranked it last.
That is the bottom row of the opening diagram, measured on the same document. Nothing about the model changed; the answer moved from rank 55 to rank 1 before generation ever ran.
4.2 Scope the context tight
Once the right subcategory is anchored, hand generation that control and its neighbours, not the whole framework and not the top-ten pages. A bounded, on-topic context removes the distractors that produce the third failure mode. Retrieval is filtering down to the smallest correct scope, not gathering the most plausible-looking material. That two-level move, anchor then size the context around it, is the whole mental model of Article 7A (retrieval is filtering, not search).
Anchor by the word that carries the question, scope to that subcategory, and generation sees the backup control and only the backup control. It is not being forced to be honest by a sterner prompt; there is simply nothing wrong in front of it.
5. The second line of defense: the answer contract
Retrieval prevents the hallucination that comes from wrong or missing context. It does not prevent all of it, and it should not try to. The generation brick carries the second line of defense: a typed answer contract that makes the model return a value, an evidence span, and a confidence, so an answer with no supporting span in the context is caught before the user sees it. Retrieval decides what the model can invent; the contract decides what it is allowed to return. Build both; they fail differently.
6. Conclusion
When a RAG system hallucinates, look upstream before you touch the prompt. Measure where the answer actually ranked. If it was not in the retrieved context, no prompt discipline will save the answer, and none of it caused the error.
Retrieval decides what the model can invent, because it decides what the model can see. The three failure modes, answer not retrieved, wrong passage retrieved, answer buried in distractors, are all retrieval conditions, and each hands generation a context it cannot answer honestly.
The fix is a retrieval fix, not a sterner prompt. Anchor on the right span with signals that survive vocabulary mismatch, then scope the context tight. On this document that one change moved the answer from rank 55 to rank 1, before generation ran. The typed answer contract is the second line of defense, not the first. Most often, hallucination in RAG is a retrieval failure logged as a generation failure.
This article is part of the Enterprise Document Intelligence series. The measurement is reproducible on the public NIST Cybersecurity Framework with a local HuggingFace embedding model, no API key required.
7. Sources and further reading
The claim that hallucination is usually a retrieval failure rests on the retrieval brick doing its job. The references below develop the fix and the second line of defense.
Earlier in the series:
The retrieval fix
The second line of defense
