A retrieval system has to answer one question over and over: which of these texts is most like this one? But most like is a human judgement, and a computer cannot compare meanings directly — it compares numbers. The bridge between those two worlds is an embedding: a function that turns a piece of text into a fixed-length list of numbers, called a vector. Once text lives as a vector, similarity becomes geometry, and geometry is something code can measure cheaply.
This lesson is the doorway to the whole retrieval arc. You will not call a model here. Instead you will build a tiny embedding by hand, watch text become numbers, and see why the location of a vector is what carries its meaning. Everything later in this module — similarity scores, a vector index, chunking — is machinery built around that single idea.
flowchart LR T["a sentence of text"] --> E["the embed function"] E --> V["a list of numbers"] V --> P["one point in vector space"] style E fill:#8b5cf6,color:#fff style P fill:#166534,color:#fff
What an embedding actually is
An embedding maps any input — a word, a sentence, a whole document — to a point in a high-dimensional space. Concretely, it produces a list of floating-point numbers: a few hundred for a small model, well over a thousand for a larger one. Each slot in the list is one dimension, and together they fix a single coordinate. The crucial property is not the numbers themselves but what the model that produced them was trained to do: place inputs that mean similar things at nearby coordinates, and inputs that mean different things far apart.
The dimension count is fixed by the model, not by the text. A two-word note and a thousand-word essay both embed to vectors of the same length — what changes is where each one lands, not how many numbers it contains.
Building one by hand
A production embedding comes from a trained model, and we will treat that model as a black box for now. But the shape of an embedding is something you can reproduce with zero training. Pick a fixed vocabulary of words. For any text, count how many times each vocabulary word appears, and write those counts as a list in vocabulary order. That list is a vector of exactly the same kind: identical length for every text, and texts that share words land close together. It is a toy, but it makes the otherwise abstract move from text to numbers completely visible.
const VOCAB = ["cat", "dog", "car", "fast"];
function embed(text, vocab) {
const words = text.toLowerCase().split(/\W+/);
return vocab.map((term) => words.filter((w) => w === term).length);
}
console.log(embed("the cat chased the dog", VOCAB));
console.log(embed("a fast car", VOCAB));
Meaning becomes location
Once two texts are vectors, the question are they similar? has a geometric answer: measure the distance between their points. Texts built from the same vocabulary words produce vectors that overlap in the same positions, so they sit close together; texts with nothing in common sit far apart. That is the whole trick in miniature. A trained embedding does the same thing across hundreds of learned dimensions and captures meaning far deeper than shared words — synonyms, paraphrases, even sentences in different languages can land near each other — but the principle underneath is identical: similar meaning, nearby point.
const VOCAB = ["cat", "dog", "pet", "car", "fast", "road"];
function embed(text, vocab) {
const words = text.toLowerCase().split(/\W+/);
return vocab.map((t) => words.filter((w) => w === t).length);
}
function dot(a, b) {
let s = 0;
for (let i = 0; i < a.length; i++) s += a[i] * b[i];
return s;
}
const animals = embed("the cat and the dog play", VOCAB);
const runs = embed("the dog runs fast", VOCAB);
const vehicle = embed("a fast car on the road", VOCAB);
console.log("animals vs runs:", dot(animals, runs)); // share 'dog'
console.log("animals vs vehicle:", dot(animals, vehicle)); // share nothing
Reading a count-vector score
The product of two count vectors has a plain reading: it counts shared content. If both texts mention dog and one mentions it twice, that overlap adds to the score; if they share no vocabulary words, the score is zero. This is crude — it rewards a word for appearing often and knows nothing about synonyms — but it is a faithful first approximation of how much do these two texts overlap?. Every fancier similarity score, including the cosine measure coming next, refines this same intuition rather than replacing it with a different idea.
flowchart TD C["nearby points"] -->|"short distance"| S["similar meaning"] F["far-apart points"] -->|"large distance"| D["different meaning"] style S fill:#166534,color:#fff style D fill:#b91c1c,color:#fff
Why so many dimensions
A bag-of-words vector has one dimension per vocabulary word, so its length grows with the dictionary you choose. A trained embedding uses a fixed, modest number of dimensions instead — but each one is a learned feature rather than a single word's raw count. Hundreds of learned dimensions give the model room to separate king from queen along one axis while keeping both far from bicycle along others. More dimensions means more room to express fine distinctions, at the cost of more storage and a slower search. That trade-off returns when you build a vector index later in this module.
flowchart LR GOOD["the word good"] -->|"shares contexts"| BAD["the word bad"] NOTE["antonyms land close together"] NOTE -.-> GOOD NOTE -.-> BAD style GOOD fill:#b45309,color:#fff style BAD fill:#b45309,color:#fff style NOTE fill:#64748b,color:#fff
const VOCAB = ["cat", "dog", "pet", "car", "fast"];
function embed(text, vocab) {
const words = text.toLowerCase().split(/\W+/);
return vocab.map((t) => words.filter((w) => w === t).length);
}
function dot(a, b) { let s = 0; for (let i = 0; i < a.length; i++) s += a[i] * b[i]; return s; }
const query = embed("my dog", VOCAB);
const docs = [
embed("the cat sleeps", VOCAB),
embed("a loyal dog pet", VOCAB),
embed("a fast car", VOCAB),
];
let best = 0;
docs.forEach((d, i) => {
const score = dot(query, d);
if (score > dot(query, docs[best])) best = i;
console.log(i, score);
});
console.log("best doc:", best);
What does this print? embed counts each vocabulary word in the text.
const VOCAB = ["cat", "dog", "car"];
function embed(text, vocab) {
const words = text.toLowerCase().split(/\W+/);
return vocab.map((t) => words.filter((w) => w === t).length);
}
console.log(embed("dog and cat", VOCAB));
Count 'cat', then 'dog', then 'car' in the text.
'dog and cat' has one cat and one dog, no car.
Write embed(text, vocab) that returns a count vector: for each vocabulary word, how many times it appears in text as a whole word (so 'category' does not count as 'cat'). Match case-insensitively.
function embed(text, vocab) {
// return one count per vocabulary word, whole-word match
}
Lowercase the text, then split on non-word characters to get whole words.
For each vocabulary term, count the words that equal it exactly.
function embed(text, vocab) {
const words = text.toLowerCase().split(/\W+/);
return vocab.map((term) => words.filter((w) => w === term.toLowerCase()).length);
}
This embed was meant to count each vocabulary word, but it uses includes, which matches substrings — so 'category' wrongly counts as one 'cat', and a word repeated twice still scores only 1. Rewrite it to match whole words and count repeats.
function embed(text, vocab) {
return vocab.map((term) => text.toLowerCase().includes(term) ? 1 : 0);
}
includes('cat') is true inside 'category' — it matches substrings, not whole words.
Split into whole words first, then count exact matches with filter.
function embed(text, vocab) {
const words = text.toLowerCase().split(/\W+/);
return vocab.map((term) => words.filter((w) => w === term).length);
}
Two text embeddings end up very close together in the vector space. What is the safest thing to conclude?
Closeness reflects statistical association learned from context, which usually tracks meaning — but antonyms and contradictions can also land close, so 'nearby' means 'used similarly', not 'agreeing' or 'identical'.
Write mostSimilar(queryText, docTexts, vocab) that returns the INDEX of the document whose count vector overlaps the query's the most (largest dot product). Include embed and dot yourself. Assume at least one document; on a tie, return the earlier index.
function embed(text, vocab) {
// your count vector
}
function dot(a, b) {
// position-by-position product, summed
}
function mostSimilar(queryText, docTexts, vocab) {
// return the index of the best-matching document
}
Embed the query once, then dot it against each embedded document.
Track the best index, replacing it only on a strictly greater score so ties keep the earlier index.
function embed(text, vocab) {
const words = text.toLowerCase().split(/\W+/);
return vocab.map((t) => words.filter((w) => w === t).length);
}
function dot(a, b) { let s = 0; for (let i = 0; i < a.length; i++) s += a[i] * b[i]; return s; }
function mostSimilar(queryText, docTexts, vocab) {
const q = embed(queryText, vocab);
const scores = docTexts.map((t) => dot(q, embed(t, vocab)));
let best = 0;
for (let i = 1; i < scores.length; i++) {
if (scores[i] > scores[best]) best = i;
}
return best;
}
Recap
- An embedding maps text to a fixed-length vector of numbers — a point in a high-dimensional space — so that similarity becomes geometry.
- Every text maps to a vector of the same length; only the location differs.
- Nearby vectors mean similar meaning; far-apart vectors mean different meaning. Retrieval is just measuring that distance.
- A bag-of-words vector is a model-free embedding you can build by hand: count each vocabulary word per text, whole-word only.
- The classic trap: embeddings encode association, not agreement or truth — antonyms like good and bad can land close, because they appear in the same contexts.
Next you will make nearby precise by implementing the actual similarity measure that retrieval systems use: cosine similarity.