Semantic search almost never crashes, and that is exactly the problem. A keyword search with no matches tells you so; a vector search always hands back its nearest neighbour, however far away that neighbour really is, and attaches a confident similarity score to it. The failure is silent: a high score, a wrong answer, no error. Feed that result to a language model as context and the model treats it as ground truth, answering with equal confidence, so the error compounds downstream and no layer beneath it ever complains.
This lesson walks through the three ways that happens for real — near-duplicate chunks, numeric or token-shaped queries, and text the model has simply never seen — and the small pieces of code that catch each one before a bad result reaches a user.
flowchart TD Q["semantic search returns a result"] --> D["score is high<br/>but possibly wrong"] D -->|"no error raised"| F["silent failure"] F --> N["near-duplicate chunks<br/>crowd the top results"] F --> T["numeric queries match<br/>the wrong year-token"] F --> O["out-of-domain text<br/>has no real answer"]
Why a high score is not a correctness guarantee
A similarity score answers one question only: how close are these two vectors? It says nothing about whether the closest vector actually answers the query. An embedding model is a map from text to geometry, built so that text which looked similar during training lands nearby. Two year-tokens like 2024 and 2023 sit as neighbours on that map, so they score highly against each other — not because one answers the other, but because they share a neighbourhood.
The model has no notion of refusing. Given any query at all, it returns its nearest point and reports the distance, and a downstream language model will read the winning chunk as reliable context because nothing in the pipeline warned it otherwise. The score is a confidence measure, not a truth measure, and reading it as truth is the mistake this lesson trains out of you.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function cos(a,b){return dot(a,b)/(mag(a)*mag(b));}
const query = [1, 1, 0];
const docs = [[1,1,0],[1,1,0],[1,0,1]];
docs.forEach((d,i)=>console.log(`doc ${i}: ${cos(query,d).toFixed(2)}`));
Failure one: near-duplicate chunks
When a document is split into chunks and several of them overlap or repeat the same sentence, every copy embeds to almost the same point. A query that matches that sentence now matches all of the copies, and they fill the top results together. The diverse document that would actually have answered the question gets pushed to rank four or lower and is never shown to the user.
The score did nothing wrong — each copy genuinely is similar to the query. Retrieval simply spent its budget on the same idea three times instead of three different ideas. A near-duplicate filter, which compares chunks to each other before search and drops the repeats, is the fix, and you will build one in the exercises.
flowchart LR Q["query"] --> K["top results"] K --> A["copy 1 of a sentence"] K --> B["copy 2 of a sentence"] K --> C["copy 3 of a sentence"] R["the genuinely useful<br/>document"] -.->|"ranked too low, unseen"| X["left out"] style A fill:#b45309,color:#fff style B fill:#b45309,color:#fff style C fill:#b45309,color:#fff style R fill:#b91c1c,color:#fff
Failure two: numeric and out-of-domain queries
Numbers, identifiers, and code tokens are the sharpest trap. The embedding model met huge numbers of numeric tokens during training and placed tokens that appeared in similar contexts close together. So a query about 2024 revenue sits right next to a chunk about 2023 revenue, and the wrong year wins with a high score. The model did not misread the year; it never read it at all. It compared geometry.
Out-of-domain text — a question in a language or jargon the model never saw — fails the same way, returning whatever happened to sit nearest. Neither failure raises an error, because from the model's point of view nothing went wrong: it was asked for a neighbour and it dutifully returned one.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function cos(a,b){return dot(a,b)/(mag(a)*mag(b));}
const query = [1, 0, 0];
const docs = [[1,0,0],[0,1,1]];
docs.forEach((d,i)=>console.log(`doc ${i}: ${cos(query,d).toFixed(2)}`));
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function safeCosine(a,b){
const ma = mag(a), mb = mag(b);
if (ma === 0 || mb === 0) return 0;
return dot(a,b)/(ma*mb);
}
function retrieve(query, docs, minScore){
let best = -1, bestScore = minScore;
docs.forEach((d,i)=>{
const s = safeCosine(query, d);
if (s > bestScore){ bestScore = s; best = i; }
});
return best;
}
console.log(safeCosine([0,0],[1,1]).toFixed(2));
console.log(retrieve([1,1,0],[[1,1,0],[0,0,1]],0.8));
console.log(retrieve([1,1,0],[[0,0,1]],0.8));
flowchart TD S["similarity score"] --> G["score above the<br/>minScore threshold"] G -->|"yes"| Y["retrieve the result"] G -->|"no"| N["return -1 and<br/>refuse to answer"] style Y fill:#166534,color:#fff style N fill:#b91c1c,color:#fff
The three guards
Each failure has a matching guard, and together they turn a raw ranking into something you can actually trust. Guard the zero vector so a similarity never collapses to NaN. Filter near-duplicate chunks before search so the top results are not three copies of one sentence. Threshold every retrieval so a low-confidence match is refused instead of returned. The runnable block above demonstrates the first and third working together, and the exercises ask you to build all three.
What does this print? It computes the cosine similarity of two vectors and formats it to three decimals.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function cosine(a,b){return dot(a,b)/(mag(a)*mag(b));}
console.log(cosine([1,0],[1,1]).toFixed(3));
dot([1,0],[1,1]) = 11 + 01 = 1.
The magnitudes are 1 and sqrt(2), so the score is 1/sqrt(2) = 0.707.
A user searches for 2024 revenue and the top result is a 2023 revenue document, returned with a high score and no error. What is the most likely failure mode?
Embedding models map tokens that appeared in similar contexts to nearby vectors, so two year-tokens land close together and score high even though the years differ. The model has no concept of refusing — it returns its nearest neighbour and reports a confident distance.
Write findNearDuplicates(docs, threshold) that returns an array of [i, j] index pairs (with i less than j) for every pair of document vectors whose cosine similarity is at least threshold. dot, mag, and cosine are already defined for you.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function cosine(a,b){return dot(a,b)/(mag(a)*mag(b));}
function findNearDuplicates(docs, threshold) {
// return [[i, j], ...] for each i < j pair with cosine(docs[i], docs[j]) >= threshold
}
Loop i from 0, and for each i loop j from i + 1, so every pair is checked once.
Push [i, j] when cosine(docs[i], docs[j]) is at least the threshold.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function cosine(a,b){return dot(a,b)/(mag(a)*mag(b));}
function findNearDuplicates(docs, threshold) {
const pairs = [];
for (let i = 0; i < docs.length; i++) {
for (let j = i + 1; j < docs.length; j++) {
if (cosine(docs[i], docs[j]) >= threshold) pairs.push([i, j]);
}
}
return pairs;
}
Write safeCosine(a, b) that returns the cosine similarity of two vectors, but returns 0 when either vector has zero magnitude. This stops an empty or all-zero chunk from producing NaN and silently poisoning the ranking. dot and mag are defined for you.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function safeCosine(a, b) {
// cosine similarity, but return 0 if either magnitude is 0
}
Compute both magnitudes first.
If either magnitude is 0, return 0 before dividing.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function safeCosine(a, b) {
const ma = mag(a), mb = mag(b);
if (ma === 0 || mb === 0) return 0;
return dot(a, b) / (ma * mb);
}
Write bestMatch(query, docs, minScore) that returns the index of the document most similar to query — but only if that best score is strictly greater than minScore. If no document clears the threshold, return -1 so the caller knows nothing was confident enough to show. On a tie for best, return the earliest index. dot, mag, and cosine are defined for you.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function cosine(a,b){return dot(a,b)/(mag(a)*mag(b));}
function bestMatch(query, docs, minScore) {
// return the best index whose score > minScore, else -1
}
Start best at -1 and bestScore at minScore, so a doc must beat the threshold to replace it.
Use a strict greater-than check so ties keep the earliest index.
function dot(a,b){let s=0;for(let i=0;i<a.length;i++)s+=a[i]*b[i];return s;}
function mag(v){return Math.sqrt(dot(v,v));}
function cosine(a,b){return dot(a,b)/(mag(a)*mag(b));}
function bestMatch(query, docs, minScore) {
let best = -1, bestScore = minScore;
for (let i = 0; i < docs.length; i++) {
const s = cosine(query, docs[i]);
if (s > bestScore) { bestScore = s; best = i; }
}
return best;
}
Recap
- Semantic search fails silently: it always returns a nearest neighbour with a confident score, even when that neighbour does not answer the query.
- A score measures confidence (how close), not correctness (does it answer) — treat it as confidence, never as truth.
- Near-duplicate chunks fill the top results with copies of one idea and crowd out the document that would have helped; filter them first.
- Numeric and out-of-domain queries match the wrong neighbour, because tokens that shared a context in training sit close together.
- Guard the zero vector: a zero magnitude makes cosine divide by zero and yield NaN, silently corrupting the ranking.
- Threshold every retrieval: if the best score is not good enough, return -1 and say so rather than guess.
Next these guards combine with an index to build a retrieval step that is fast, correct, and honest about what it does not know.