The retriever you built in the last lesson returns the chunks most similar to a query. That sounds right until you notice what similarity actually rewards: a chunk that paraphrases the question back at you shares many of its words, so it scores highly on cosine similarity, even though it contributes nothing the user did not already say. It can quietly crowd out the chunk that genuinely answers. This is the central weakness of single-stage retrieval, and it is the gap a second stage exists to close. Retrieve-then-rerank is the standard fix: cast a wide, cheap net first, then re-score that shortlist with something more discerning.
flowchart LR Q["query"] --> R["stage 1 retrieve<br/>top-N by embedding cosine"] CORPUS["the whole corpus"] --> R R --> SHORT["shortlist of N candidates"] SHORT --> RR["stage 2 rerank<br/>top-k by a precise scorer"] RR --> TOP["final top-k context"] style R fill:#b45309,color:#fff style RR fill:#8b5cf6,color:#fff
Two stages, two goals
Stage one is the retriever. It is fast and cheap — a cosine search over the whole index — and its job is recall: do not miss the relevant chunks, even if some junk slips in. It returns a broad shortlist of, say, the top twenty. Stage two is the reranker. It is slower and more expensive, because it scores each candidate against the query jointly rather than by vector angle, but it is far more discerning. Its job is precision at the top: of the shortlist, push the truly useful chunks to the front and return the final top k. You trade a little extra latency for a much better-chosen context.
Why not just rerank everything?
The honest reason is cost. A cross-encoder reranker scores one query-and-chunk pair at a time, and that score is not cheap to compute. Running it across a million-document corpus for every question would be ruinously slow. The retriever, by contrast, is an approximate nearest-neighbour lookup that costs a few milliseconds no matter how large the index grows. So you let the cheap, broad stage narrow a million documents down to twenty, and only then spend the expensive stage's judgement on those twenty. The split is not arbitrary elegance — it is the only way an accurate scorer stays affordable at scale.
flowchart TD R["retriever: cheap and broad<br/>optimises recall<br/>scores every document"] -->|"returns top-N"| N["broad shortlist<br/>few missed, some junk"] N --> X["reranker: expensive and precise<br/>optimises precision<br/>scores only the shortlist"] X -->|"returns top-k"| K["tight, accurate top-k"] style R fill:#166534,color:#fff style X fill:#8b5cf6,color:#fff
Why the reranker is more discerning
A retriever embeds the query and each chunk separately, then compares the finished vectors — it never sees the two texts together. A reranker is usually a cross-encoder: it reads the query and the candidate chunk as one combined input and scores how well the chunk actually answers that specific question. That joint reading is what lets it tell a real answer from a clever paraphrase, and it is also exactly why it is slower — there is no pre-computed index to look up, only a fresh scoring pass over each candidate pair.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function sim(a,b){const m=mag(a)*mag(b);return m===0?0:dot(a,b)/m;}
const docs = [
{ id: "answer", vec: [1,1,0], score: 0.95 },
{ id: "paraphrase", vec: [1,1,1], score: 0.20 },
{ id: "unrelated", vec: [0,0,1], score: 0.05 }
];
const query = [1,1,1];
const retrieved = docs.slice()
.map(d => ({ d, s: sim(query, d.vec) }))
.sort((a,b) => b.s - a.s)
.map(x => x.d.id);
console.log("retrieve order:", retrieved);
const reranked = retrieved
.map(id => docs.find(d => d.id === id))
.sort((a,b) => b.score - a.score)
.map(d => d.id);
console.log("rerank order: ", reranked);
The reordering, made visible
The example above stages a retrieval and a rerank over the same three chunks and prints both orderings. Watch the top: the chunk that wins on embedding cosine is a paraphrase of the question, but the reranker — here a stand-in score representing what a real cross-encoder would output — puts the actual answer first. Nothing about the chunks changed between the two lines; only the scoring did. That flip, from most similar to most useful, is the entire payoff of the second stage. The retrieved shortlist kept the answer available; the reranker's job was simply to recognise it.
flowchart TD A["after retrieve<br/>top is the most SIMILAR chunk"] --> RR["reranker re-scores<br/>for genuine relevance"] RR --> B["after rerank<br/>top is the most USEFUL chunk"] style A fill:#b91c1c,color:#fff style B fill:#166534,color:#fff
function precisionAtK(returnedIds, relevantSet, k) {
const top = returnedIds.slice(0, k);
const hits = top.filter(id => relevantSet.has(id)).length;
return hits / k;
}
const relevant = new Set(["answer"]);
const retrieveOrder = ["paraphrase", "answer", "unrelated"];
const rerankOrder = ["answer", "paraphrase", "unrelated"];
console.log("retrieve precision@1:", precisionAtK(retrieveOrder, relevant, 1));
console.log("rerank precision@1:", precisionAtK(rerankOrder, relevant, 1));
Measuring the improvement
How do you know reranking helped, rather than just feeling that the answers got better? You measure precision at k: of the top k chunks returned, what fraction are truly relevant? The example above labels one chunk as the real answer and computes precision at one for the retrieve-only order and the reranked order. Retrieve-only puts the paraphrase first and scores zero; rerank puts the answer first and scores one. A number, not an opinion. The next module's evaluation lesson generalises this into recall@k and precision@k over labelled fixtures — the disciplined way to decide whether a retrieval change is actually an improvement.
What does this print? rerank returns the items sorted by the score function, highest first.
function rerank(items, score) {
return items.slice().sort((a, b) => score(b) - score(a));
}
const oracle = { a: 0.2, b: 0.9, c: 0.5 };
const items = [{ id: "a" }, { id: "b" }, { id: "c" }];
console.log(rerank(items, x => oracle[x.id]).map(x => x.id));
Score each item: a is 0.2, b is 0.9, c is 0.5.
Highest score first: b, then c, then a.
Write rerank(items, score) that returns a NEW array of the items sorted by score descending (most relevant first), WITHOUT mutating the original items array.
function rerank(items, score) {
// return a new array, sorted by score descending; leave items untouched
}
Copy the array first with slice, so sort does not touch the original.
A descending sort subtracts the earlier score from the later one: score(b) - score(a).
function rerank(items, score) {
return items.slice().sort((a, b) => score(b) - score(a));
}
Write retrieveThenRerank(query, docs, scoreFn, n, k) for docs of shape {id, vec}. Stage one: retrieve the top-n docs by cosine similarity to query. Stage two: rerank those n by scoreFn and return the top-k ids. Include the cosine helpers yourself.
function dot(a, b) {
}
function mag(v) {
}
function sim(a, b) {
}
function retrieveThenRerank(query, docs, scoreFn, n, k) {
// retrieve top-n by cosine, rerank those by scoreFn, return top-k ids
}
Map each doc to {d, s: sim(query, d.vec)}, sort by s descending, slice to n for the shortlist.
Then sort the shortlist by scoreFn descending and slice to k for the final answer ids.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function sim(a,b){const m=mag(a)*mag(b);return m===0?0:dot(a,b)/m;}
function retrieveThenRerank(query, docs, scoreFn, n, k) {
const scored = docs.map(d => ({ d, s: sim(query, d.vec) }));
scored.sort((a, b) => b.s - a.s);
const retrieved = scored.slice(0, n).map(x => x.d);
const reranked = retrieved.slice().sort((a, b) => scoreFn(b) - scoreFn(a));
return reranked.slice(0, k).map(d => d.id);
}
This retrieveThenRerank is supposed to rerank only the top-n RETRIEVED candidates, but it reranks ALL documents instead — defeating the two-stage design and returning chunks the retriever never selected. Fix it so the reranker only sees the retrieved top-n.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function sim(a,b){const m=mag(a)*mag(b);return m===0?0:dot(a,b)/m;}
function retrieveThenRerank(query, docs, scoreFn, n, k) {
const reranked = docs.slice().sort((a, b) => scoreFn(b) - scoreFn(a));
return reranked.slice(0, k).map(d => d.id);
}
First score every doc by cosine similarity and slice to the top n — that is the reranker's input.
Then sort only that shortlist by scoreFn and slice to k. Reranking the whole corpus skips stage one entirely.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function sim(a,b){const m=mag(a)*mag(b);return m===0?0:dot(a,b)/m;}
function retrieveThenRerank(query, docs, scoreFn, n, k) {
const scored = docs.map(d => ({ d, s: sim(query, d.vec) }));
scored.sort((a, b) => b.s - a.s);
const retrieved = scored.slice(0, n).map(x => x.d);
const reranked = retrieved.slice().sort((a, b) => scoreFn(b) - scoreFn(a));
return reranked.slice(0, k).map(d => d.id);
}
Why does a RAG pipeline retrieve a broad set first and rerank a narrow set second, rather than running the accurate reranker over every document?
A cross-encoder reranker scores each query-and-chunk pair jointly, which is too expensive to run across millions of documents per query. The cheap retriever gets high recall by narrowing to a shortlist the reranker can then afford to score precisely.
Recap
- Single-stage retrieval confuses similarity with usefulness — a paraphrase of the question scores high on cosine but adds nothing.
- Retrieve-then-rerank splits the job: a cheap retriever optimises recall (a broad shortlist), an expensive reranker optimises precision at the top.
- The split is driven by cost: the reranker is too slow for the whole corpus, so the retriever narrows it to a shortlist first.
- A reranker is usually a cross-encoder that reads query and chunk together, which is why it is more discerning and also slower than separate-vector cosine.
- Reranking re-sorts the shortlist, often flipping a similar-but-useless chunk out of the top for the real answer.
- It cannot rescue a miss: if the retriever never returned the right chunk, reranking cannot find it — so recall is the foundation, precision the polish.
- Measure the gain with precision@k against a labelled set, not by feel.