You send a model a thousand words and the invoice comes back billed in tokens, not words — and the two numbers almost never agree. The reason is straightforward once you see it: the model never reads your words. Before a single weight fires, your text is sliced into pieces called tokens by a tokenizer, and tokens are the unit everything downstream is measured in. Price is per token. The context window is counted in tokens. The length limit that throws a 400 error is a token limit. This lesson is about why that unit exists and how to reason about it without spending a cent on an API call. Once you can count tokens yourself, two everyday mysteries dissolve at once: why a bill does not match your word count, and why a document you considered short gets rejected at the door for being too long.
Three units, prized apart
Three units get constantly conflated and have to be separated. A character is a single symbol — the c in cat. A word is whatever a space delimits — cat is one word, and so is catnap. A token is neither: it is whatever chunk a particular tokenizer decided to treat as one unit, and those decisions were learned from a training corpus rather than read off the page. The defining property is that the mapping from word to token is not one-to-one. A frequent word like cat is almost always a single token, but a long or unusual word is broken into several subword tokens — tokenization may become token plus isation. That gap, between one token and several, is the whole reason token count drifts away from word count, and which side a word falls on depends entirely on the tokenizer's vocabulary.
flowchart LR T["your text"] --> K["tokenizer"] K --> S["token token token"] S --> I["id id id"] style K fill:#3776ab,color:#fff
Why split words at all
You might wonder why the tokenizer bothers splitting words instead of handing each whole word its own token. A vocabulary of every possible word would be unbounded — typos, names, and newly coined terms would have no representation — while a vocabulary of single characters would make every sentence hundreds of tokens long and destroy the model's sense of what a word even is. Subword tokenization threads between the two. Frequent, complete words get a single token so common text stays compact; everything else is assembled from reusable pieces, so even a word the tokenizer has never once seen can be spelled out from parts it does know. The vocabulary is therefore a fixed, finite list the model was trained against, and the very same sentence can produce a different token count under a different model, because each model ships its own tokenizer with its own learned vocabulary.
const vocab = { the: 1, cat: 2, sat: 3, on: 4, mat: 5 };
function tokensFor(word) {
return word in vocab ? 1 : 2; // known -> 1 token, unknown -> splits into 2
}
function countTokens(text) {
return text.split(" ").reduce((sum, w) => sum + tokensFor(w), 0);
}
console.log(countTokens("the cat sat")); // 3 known words -> 3
console.log(countTokens("the cat xyzzy")); // xyzzy unknown -> 1 + 1 + 2 = 4
Three counts, three numbers
Here is the punchline in a single line: for any non-trivial text, word count, character count, and token count are three different numbers, and only the last one ever appears on a bill or in an error message. A ten-word sentence is not ten tokens — punctuation, spaces, and subword splits all nudge the total — and it is certainly not equal to its character count. When a provider says a model has a 128,000-token context window, that window is filled in tokens, so the only honest way to know how much room you have left is to count tokens the same way the provider does. Estimating from words is a rounding error that quietly becomes a production outage once you are operating at scale.
flowchart TD C["cat"] --> O1["1 token"] TN["tokenization"] --> O2["token + isation = 2 tokens"] X["xyzzy"] --> O3["x + y + z + z + y = 5 tokens"] style O1 fill:#166534,color:#fff style O2 fill:#b45309,color:#fff style O3 fill:#b91c1c,color:#fff
const text = "the cat sat on the mat";
console.log("words:", text.split(" ").length);
console.log("chars:", text.length);
const vocab = { the: 1, cat: 2, sat: 3, on: 4, mat: 5 };
const tokens = text.split(" ").reduce((n, w) => n + (w in vocab ? 1 : 2), 0);
console.log("tokens:", tokens);
The boundary trap
The first surprise that catches engineering teams is that whitespace and punctuation are frequently tokens in their own right, and that the same word can tokenize differently depending on what sits beside it. A leading space is often absorbed into the following token, so cat and cat are not guaranteed to be the same token. Contractions like can't are typically two tokens, not one. Capitalisation also matters: The and the may map to different ids entirely. The practical consequence is the one that bites in production — concatenating two strings and then tokenizing the joined result is not the same as tokenizing each piece separately and adding the two counts. Anyone who builds a prompt by gluing fragments together and budgets tokens per fragment learns this the expensive way.
flowchart LR S["one sentence"] --> W["words: 6"] S --> H["chars: 22"] S --> K["tokens: 6 here, different elsewhere"] style K fill:#8b5cf6,color:#fff
function roughTokens(s) {
const pieces = s.match(/[a-z]+|[^a-z ]/gi) || [];
return pieces.length;
}
console.log(roughTokens("the cat sat")); // 3
console.log(roughTokens("the cat's sat")); // the cat ' s sat -> 5
console.log(roughTokens("the cat sat.")); // adds the period -> 4
What does this print? Words found in vocab cost 1 token; any word not in vocab is treated as split subwords and costs 2.
const vocab = { the: 1, cat: 2, sat: 3 };
function tokensFor(word) { return word in vocab ? 1 : 2; }
function countTokens(text) {
return text.split(" ").reduce((sum, w) => sum + tokensFor(w), 0);
}
console.log(countTokens("the cat sat xyz"));
the, cat, sat are all in vocab, so each costs 1 token — that is 3 so far.
xyz is not in vocab, so it costs 2. Total is 3 + 2.
Write tokenize(text, vocab) that splits text on single spaces, looks each word up in vocab (a map of word to token id), and returns an array of the ids in order. Skip any word that is not in vocab — emit nothing for it.
function tokenize(text, vocab) {
// split on spaces, map known words to their id, skip unknowns
}
Loop over text.split(" ") and check
w in vocabfor each word.Push vocab[w] when the word is known; otherwise push nothing.
function tokenize(text, vocab) {
const ids = [];
for (const w of text.split(" ")) {
if (w in vocab) ids.push(vocab[w]);
}
return ids;
}
In vocab, a common word maps to a single token id (a number), but a rare word maps to an array of ids because it was split into subwords. totalTokens should return the total number of token ids across the whole text. Right now it counts every known word as exactly one token, so a word mapped to [10, 11] is undercounted. Fix it so a number counts as 1 and an array counts as its length.
function totalTokens(text, vocab) {
let n = 0;
for (const w of text.split(" ")) {
if (w in vocab) n += 1; // BUG: ignores that a value may be an array
}
return n;
}
Each word's value is either a number (1 token) or an array (as many tokens as its length).
Use Array.isArray(v) to tell them apart, and add v.length when it is an array.
function totalTokens(text, vocab) {
let n = 0;
for (const w of text.split(" ")) {
if (w in vocab) {
const v = vocab[w];
n += Array.isArray(v) ? v.length : 1;
}
}
return n;
}
Write fewerTokens(a, b, vocab) that returns whichever of a or b tokenises to fewer token ids (using tokenize, provided for you). On a tie, return a.
function tokenize(text, vocab) {
const ids = [];
for (const w of text.split(" ")) {
if (w in vocab) ids.push(vocab[w]);
}
return ids;
}
function fewerTokens(a, b, vocab) {
// return a or b, whichever produces fewer ids (tie -> a)
}
Call tokenize on both strings and compare their .length.
Return a when la <= lb, so that a wins a tie.
function tokenize(text, vocab) {
const ids = [];
for (const w of text.split(" ")) {
if (w in vocab) ids.push(vocab[w]);
}
return ids;
}
function fewerTokens(a, b, vocab) {
const la = tokenize(a, vocab).length;
const lb = tokenize(b, vocab).length;
return la <= lb ? a : b;
}
Why can the same word cost a different number of tokens under two different models?
A token's boundaries are decided by the tokenizer's vocabulary, which is learned per model. A word that is a single learned token in one vocabulary may need to be spelled out from subwords in another, so identical text yields different token counts across models.
Recap
- The unit a model is priced on and limited by is the token, not the word and not the character.
- A token is whatever chunk a tokenizer treats as one unit; the word-to-token mapping is learned and is not one-to-one.
- Frequent words are usually one token; rare or long words split into several subword tokens — which is why token count diverges from word count.
- Word count, character count, and token count are three different numbers; only token count appears on a bill or in an error.
- Whitespace, punctuation, and capitalisation are often tokens themselves — so when accuracy matters, tokenise the final assembled string, not the fragments.
- Each model has its own vocabulary, so the same text can cost a different number of tokens under a different model.
Next you will open the tokenizer up and build a byte-pair-encoding merge loop from scratch.