You changed the chunk size. Or you swapped embedding models. Or you raised top-k from five to ten. The obvious question — did retrieval get better? — has no obvious answer. Staring at the top results and deciding they 'look about right' is not an evaluation; it is a feeling, and feelings are how a retriever quietly regresses for months while everyone insists the new setup is fine. To know whether a change helped, you need the same instrument every search engine has used for decades: a small set of queries where you already know the right answer, and two numbers that say how much of the right answer you found and how much junk came along with it.
The setup is a labelled fixture: a list of example queries, and for each one, the set of chunk ids that truly answer it. That ground-truth set is assembled by hand, usually once, and it is the single most valuable artefact in a retrieval project — far more valuable than any embedding model, because it is the thing that lets you tell good from bad. A retriever is then run over the queries, returning a ranked list of chunk ids per query. Evaluation is just comparing what came back against what should have come back. Everything in this lesson is deterministic and needs no model: the fixture is fixed, the ranking is fixed, and the arithmetic is a handful of set operations.
flowchart LR F["labelled fixture<br/>queries + known-relevant ids"] --> RT["run the retriever<br/>get ranked results"] RT --> CMP["compare top-k<br/>against the relevant set"] CMP --> M["recall@k<br/>and precision@k"] M --> DEC["keep the change,<br/>or revert it"] style M fill:#8b5cf6,color:#fff style DEC fill:#166534,color:#fff
Recall@k measures coverage: of all the chunks that were relevant, what fraction did your top-k include? You take the intersection of the retrieved top-k with the relevant set and divide by the size of the relevant set. If three chunks truly answer a query and your top-k contains two of them, recall@k is two thirds. Recall is the metric for the missing chunk failure from the previous lesson: every relevant chunk you failed to surface drags recall down. Its denominator is the relevant set, not k — a small detail that is easy to get wrong, and the exercises will press on exactly that point. A recall of one means you found everything that mattered, no matter how much junk came with it.
flowchart LR REL["relevant: 2, 4, 7"] -->|"2 found"| INT["intersection<br/>size 2"] RET["top-2: 4, 2"] --> INT INT --> R["recall 2/3<br/>precision 2/2"] style INT fill:#8b5cf6,color:#fff style R fill:#166534,color:#fff
function recallAtK(retrieved, relevant, k) {
const top = retrieved.slice(0, k);
const hits = top.filter(id => relevant.includes(id)).length;
return relevant.length === 0 ? 0 : hits / relevant.length;
}
function precisionAtK(retrieved, relevant, k) {
const top = retrieved.slice(0, k);
const hits = top.filter(id => relevant.includes(id)).length;
return k === 0 ? 0 : hits / k;
}
const retrieved = [10, 30, 20, 40];
const relevant = [10, 20];
console.log("recall@2 =", recallAtK(retrieved, relevant, 2));
console.log("precision@2 =", precisionAtK(retrieved, relevant, 2));
Precision@k measures purity: of the slots your top-k filled, what fraction were actually relevant? It uses the same intersection, but divides by k, the number of slots you filled. If your top-five holds two relevant chunks and three pieces of junk, precision@k is two fifths. Precision is the metric for the wrong chunk failure: every irrelevant chunk that crowded into the top-k lowers it. Notice the asymmetry — recall counts missed good chunks, precision counts admitted bad ones. A change that raises k will usually lift recall (more chances to hit a good chunk) while lowering precision (more room for junk). That tension is exactly why you measure both, never one alone.
flowchart TD K1["k=1: recall 0.33, precision 1.0"] --> K2["k=2: recall 0.67, precision 1.0"] K2 --> K3["k=3: recall 1.0, precision 1.0"] K3 --> K4["k=4: recall 1.0, precision 0.75"] K4 --> K5["k=5: recall 1.0, precision 0.6"] K5 --> NOTE["higher k lifts recall,<br/>then precision falls as junk enters"] style NOTE fill:#b45309,color:#fff
function recallAtK(retrieved, relevant, k) {
const top = retrieved.slice(0, k);
const hits = top.filter(id => relevant.includes(id)).length;
return relevant.length === 0 ? 0 : hits / relevant.length;
}
function precisionAtK(retrieved, relevant, k) {
const top = retrieved.slice(0, k);
const hits = top.filter(id => relevant.includes(id)).length;
return k === 0 ? 0 : hits / k;
}
const retrieved = [4, 2, 7, 1, 3];
const relevant = [2, 4, 7];
for (let k = 1; k <= 5; k++) {
console.log("k=" + k, "recall=" + recallAtK(retrieved, relevant, k).toFixed(2), "precision=" + precisionAtK(retrieved, relevant, k).toFixed(2));
}
Run the numbers and the trade-off becomes visible: as k climbs, recall rises toward one while precision falls away. Neither number is 'the score'; a retriever is a balance between them. Two pitfalls catch teams here. The first is evaluating on the same few queries you tuned against — you will overfit the retriever to those examples and ship a number that does not hold up on real traffic. The second is a fixture so small, or so easy, that every variant scores perfectly, so the metric cannot tell a real improvement from noise. A fixture earns its keep by containing queries your current retriever actually gets wrong.
function recallAtK(retrieved, relevant, k) {
const top = retrieved.slice(0, k);
const hits = top.filter(id => relevant.includes(id)).length;
return relevant.length === 0 ? 0 : hits / relevant.length;
}
function meanRecall(fixtures) {
let sum = 0;
for (const f of fixtures) sum += recallAtK(f.retrieved, f.relevant, f.k);
return sum / fixtures.length;
}
const fixtures = [
{ retrieved: [1, 2, 3], relevant: [1, 2], k: 2 },
{ retrieved: [1, 2], relevant: [2, 3], k: 2 }
];
console.log("mean recall =", meanRecall(fixtures));
What does this print? Only one of the two relevant ids appears in the top-2.
function precisionAtK(retrieved, relevant, k) {
const top = retrieved.slice(0, k);
const hits = top.filter(id => relevant.includes(id)).length;
return hits / k;
}
console.log(precisionAtK([10, 20, 30], [20, 40], 2));
The top-2 is [10, 20].
Only 20 is in the relevant set [20, 40], so there is 1 hit.
Precision divides hits by k: 1 / 2 = 0.5.
Write recallAtK(retrieved, relevant, k): the fraction of the relevant chunk ids that appear in the first k entries of retrieved. Return 0 when relevant is empty (nothing relevant means nothing was missed). Both arguments are arrays of chunk ids.
function recallAtK(retrieved, relevant, k) {
// hits in the top-k, divided by the number of relevant ids
}
Take the first k entries with
retrieved.slice(0, k).Count how many of those are in
relevantusing.filterand.includes.Divide the hit count by
relevant.length— and guard the empty case.
function recallAtK(retrieved, relevant, k) {
const top = retrieved.slice(0, k);
const hits = top.filter(id => relevant.includes(id)).length;
return relevant.length === 0 ? 0 : hits / relevant.length;
}
Write precisionAtK(retrieved, relevant, k): the fraction of the first k entries of retrieved that are in the relevant set. Return 0 when k is 0. You may assume retrieved has at least k entries.
function precisionAtK(retrieved, relevant, k) {
// hits in the top-k, divided by k
}
Same intersection as recall, but divide by k, not by the relevant set.
Count hits in the top-k, then divide by k (guard k === 0).
function precisionAtK(retrieved, relevant, k) {
const top = retrieved.slice(0, k);
const hits = top.filter(id => relevant.includes(id)).length;
return k === 0 ? 0 : hits / k;
}
This recallAtK is supposed to return the fraction of the relevant set found in the top-k, but it divides by k instead of by the size of the relevant set. That makes recall look better the more junk you retrieve — exactly backwards. Fix the denominator, and keep the empty-relevant guard returning 0.
function recallAtK(retrieved, relevant, k) {
const top = retrieved.slice(0, k);
const hits = top.filter(id => relevant.includes(id)).length;
return relevant.length === 0 ? 0 : hits / k;
}
Recall's denominator is the number of relevant ids, not k.
relevant.lengthis the size of the relevant set.
function recallAtK(retrieved, relevant, k) {
const top = retrieved.slice(0, k);
const hits = top.filter(id => relevant.includes(id)).length;
return relevant.length === 0 ? 0 : hits / relevant.length;
}
Write meanRecall(fixtures) that averages recall@k across several queries. Each entry in fixtures is an object { retrieved, relevant, k }. A query with an empty relevant contributes 0. Assume fixtures is non-empty. Reimplement the recall formula inside your function — each exercise runs in a fresh interpreter that cannot see earlier code.
function meanRecall(fixtures) {
// average the recall@k of every fixture
}
Loop over the fixtures, computing recall@k for each.
Accumulate a running sum and divide by the number of fixtures.
Reimplement the recall formula inside the loop — do not assume a helper exists.
function meanRecall(fixtures) {
let sum = 0;
for (const f of fixtures) {
const top = f.retrieved.slice(0, f.k);
const hits = top.filter(id => f.relevant.includes(id)).length;
const r = f.relevant.length === 0 ? 0 : hits / f.relevant.length;
sum += r;
}
return sum / fixtures.length;
}
Recap
- 'It feels better' is not an evaluation. A labelled fixture (queries with known-relevant ids) is the instrument that turns a feeling into a number.
- Recall@k = relevant found in top-k, divided by the size of the relevant set. It measures coverage and catches the missing chunk.
- Precision@k = relevant found in top-k, divided by k. It measures purity and catches the wrong chunk.
- Raising k lifts recall and lowers precision, so measure both — never one alone.
- Mind the denominator: recall divides by the relevant set, precision by k. Mixing them up makes junk look like success.
Next you will wire these metrics into a retrieve-then-rerank flow and watch reranking move the numbers.