Retrieval-augmented generation looks unbeatable on the demo. Paste in your documents, ask a question, and out comes a fluent answer with a citation attached. The trouble begins the moment the demo ends. A RAG pipeline has no judgement of its own: it takes whatever chunks the retriever happened to return and hands them to a model whose single great talent is sounding sure of itself. Give it the right passage and it shines. Give it nothing, or the wrong thing, and it will still answer — fluently, confidently, and wrong.
That is the central failure of naive RAG, and it is almost never the model that breaks. The model does exactly what models do. What breaks is the retrieval step, silently handing the model garbage while nothing in the pipeline notices. This lesson reproduces the three ways that happens and the cheap, deterministic guards that catch each one. The missing chunk is when the answer exists in your corpus but retrieval never surfaced it. The wrong chunk is when something irrelevant but high-scoring sneaks into the context. The contradictory chunks case is when two passages both look relevant but disagree. In all three, the model happily answers anyway.
flowchart TD Q["user query"] --> R["retriever<br/>returns top-k"] R --> M["missing chunk<br/>answer not in top-k"] R --> W["wrong chunk<br/>relevant-looking, off-topic"] R --> X["contradictory chunks<br/>two sources disagree"] M --> H["model answers anyway<br/>confident nonsense"] W --> H X --> H style H fill:#b91c1c,color:#fff
The missing chunk is the sneakiest of the three because the system reports success. The user asks about the refund window; the real refund policy is sitting right there in your vector store; but the chunk that holds it scored 0.41 and ranked just outside the top-k, so it never reached the model. With no relevant passage in context, the model does what it does with any prompt — it generates the most likely-sounding next tokens. Those tokens happen to describe a refund window, plausibly stated, with no source attached. Nothing errored. Nothing refused. The failure is invisible unless you go looking for it, which is exactly why a later lesson is devoted to measuring retrieval rather than trusting it.
function stubModel(contextChunks) {
const context = contextChunks.join(" ");
return "Based on the provided context: " + (context || "(none given)");
}
const query = "what is the refund window?";
const retrieved = [];
console.log("answer:", stubModel(retrieved));
console.log("refused?", retrieved.length === 0);
The wrong chunk is the mirror image: retrieval returned plenty, just not the right thing. A question about refunds pulls in a high-scoring passage about returns, because the two words sit near each other in embedding space, and the model dutifully answers the question it was handed evidence for — which was not the one the user asked. Contradictory chunks are nastier still. Your corpus holds a 2024 policy saying the window is 14 days and a 2026 policy saying 30. Both get retrieved. Both look authoritative. The model picks one, or quietly invents a compromise like 'around 21 days' that appears in neither source. A sourced answer that is half right is more dangerous than an honest refusal, because the citation makes the wrong part look verified.
flowchart LR
Q["query"] --> SC["score every chunk"]
SC --> TH{"best score<br/>above threshold"}
TH -->|"no"| RF["refuse:<br/>no confident context"]
TH -->|"yes"| AN["answer with<br/>citations"]
style RF fill:#b45309,color:#fff
style AN fill:#166534,color:#fff
const chunks = [
{ id: "a", text: "refunds issued within 30 days", score: 0.82 },
{ id: "b", text: "our office is closed on sunday", score: 0.31 },
{ id: "c", text: "returns require a receipt", score: 0.74 },
{ id: "d", text: "the lunch menu updates weekly", score: 0.22 }
];
const THRESHOLD = 0.5;
const relevant = chunks.filter(c => c.score >= THRESHOLD);
console.log("kept:", relevant.map(c => c.id).join(","));
console.log("safe to answer?", relevant.length > 0);
The guard against all three failure modes is the same shape: before the model ever sees context, score every chunk and keep only what clears a relevance threshold. If nothing clears it, the pipeline must refuse rather than hand the model an empty stage. That refusal is the whole game — it is what turns a confident hallucination into an honest 'I cannot find that in the sources'. But the guard itself has a famous trap, and it lives in a single line of JavaScript, so it is worth building carefully. The next block reproduces the contradiction check; the callout after it dismantles the threshold bug.
function hasConflict(chunks) {
const values = new Set(chunks.map(c => c.value));
return values.size > 1;
}
const retrieved = [
{ source: "policy-2024.pdf", value: 14 },
{ source: "policy-2026.pdf", value: 30 }
];
console.log("contradicts?", hasConflict(retrieved));
console.log("model would have to pick:", retrieved[0].value, "or", retrieved[1].value);
flowchart LR
ANS["model answer<br/>claim1, claim2, claim3"] --> CHK{"each claim<br/>found in a source?"}
CHK -->|"yes"| OK["grounded<br/>cite the source"]
CHK -->|"no"| BAD["ungrounded<br/>flag or refuse"]
style OK fill:#166534,color:#fff
style BAD fill:#b91c1c,color:#fff
The last guard is grounding. Even when retrieval returns strong chunks and the model answers, every claim in that answer ought to trace back to a chunk you actually retrieved. If the model asserts a number, a date, or a policy that appears in none of the sources, that claim is ungrounded — invented, however confidently it is stated. Catching that means checking each claim word against the source text. It is crude, and production systems use a model to do it more cleverly, but the core idea is exactly this deterministic membership check: if a claim is not present in the sources, flag it. The exercise below asks you to build that check.
What does this print? The detector returns true when the retrieved chunks assert more than one distinct value.
function contradicts(chunks) {
const vals = new Set(chunks.map(c => c.value));
return vals.size > 1;
}
console.log(contradicts([{value: 7}, {value: 7}, {value: 9}]));
Build a Set of the chunk values: 7, 7, and 9.
A Set drops duplicates, so it holds {7, 9}. Its size is 2.
Two distinct values means a contradiction, so the function returns true.
Write filterRelevant(chunks, threshold) that returns only the chunks whose score is greater than or equal to threshold. This is the guard that keeps low-scoring noise out of the model's context.
function filterRelevant(chunks, threshold) {
// return only chunks with score >= threshold
}
Use the array
.filtermethod.Keep a chunk when its
scoreis>= threshold.
function filterRelevant(chunks, threshold) {
return chunks.filter(c => c.score >= threshold);
}
This refusal gate is meant to return true only when at least one retrieved chunk clears the relevance threshold, so the model answers only from real evidence. It uses .every, which is wrong on two counts: it demands that ALL chunks clear the bar, and worse, it returns true for an empty retrieved set, so the model answers with no context at all. Fix it so it returns true when ANY chunk clears the threshold.
function shouldAnswer(chunks, threshold) {
return chunks.every(c => c.score >= threshold);
}
.everyis true when all elements pass, and vacuously true for an empty array.You want true when at least one passes. Reach for
.some, which is false on an empty array.
function shouldAnswer(chunks, threshold) {
return chunks.some(c => c.score >= threshold);
}
Write ungroundedClaims(answer, sources) that returns the list of claim words from answer that appear in NONE of the sources strings. Here answer is a single space-separated string of words, and sources is an array of source-text strings. A claim word found in no source is ungrounded. Split on spaces and compare case-sensitively.
function ungroundedClaims(answer, sources) {
// return answer words that appear in no source
}
Build a Set of every word across all source strings: join them, then split on spaces.
Filter the answer's words, keeping only those the Set does NOT have.
function ungroundedClaims(answer, sources) {
const known = new Set(sources.join(" ").split(" "));
return answer.split(" ").filter(w => !known.has(w));
}
A support bot is asked 'how long is the refund window?'. The retriever returns three chunks, none about refunds — the real refund-policy chunk scored 0.41 and ranked fourth, just outside the top-3. The model answers '30 days' anyway. Which failure mode is this?
The refund policy was in the corpus (it scored 0.41) but sat just outside the top-k, so the model answered from no relevant evidence. That is the missing-chunk failure — low recall — and the fix is to raise k, lower the threshold, or improve the embedding, not to trust the fluent answer.
Recap
- Naive RAG fails silently because the model never refuses on its own — the pipeline must catch empty or conflicting context before it ever reaches the model.
- The three failure modes are the missing chunk (low recall), the wrong chunk (low precision), and contradictory chunks (no freshness or dedup guard).
- Guard with a relevance threshold: keep chunks above it, and refuse to answer when none clear the bar.
- Watch the
.everyon an empty array trap — use.someso the gate fails closed instead of open. - Enforce grounding: every claim in the answer must appear in a retrieved source, or it is flagged as ungrounded.
Next you will measure retrieval itself with recall and precision, so you can tell a real fix from a feeling.