The cheapest request is the one you never make. Once you know where the tokens go, the next question is whether you are paying to compute an answer you have already computed before. Most real workloads are astonishingly repetitive: the same FAQs, the same greetings, the same templated summaries. A system that re-runs the model for each one is burning money to reproduce an identical string.
A cache fixes this. Before calling the model, look the request up in a store; if the answer is already there, return it and skip the call entirely. The trade is memory for money, and for text generation it is almost always worth it. This lesson builds the cache, the part that decides whether two requests count as 'the same', and the measurement that tells you whether any of it is actually working. All deterministic, no model required.
The lookup, in the small
A cache is a map from a key to a value, plus one rule for using it. On a request you compute the key and check the map: if the key is present it is a hit — return the stored value and do no further work. If it is absent it is a miss — run the expensive computation, store its result under the key, and return that. The next identical request becomes a hit.
The whole strategy lives in that one branch. Every hit is a model call you did not make, which is tokens you did not buy and latency you did not wait for. The computation you cache can be anything; here it stands in for a model call with a little counter, so you can see the calls disappear. Notice the asymmetry: a miss still costs full price, but it pays once for every future hit it buys.
const cache = new Map();
function lookup(key, make) {
if (cache.has(key)) return cache.get(key); // hit: reuse
const value = make(); // miss: compute once
cache.set(key, value);
return value;
}
let calls = 0;
const make = () => ++calls * 10; // stands in for an expensive model call
lookup('q1', make); // miss -> calls=1, stores 10
lookup('q1', make); // hit -> reused, no call
lookup('q2', make); // miss -> calls=2, stores 20
console.log('calls =', calls);
console.log('q1 ->', lookup('q1', make)); // hit -> 10
flowchart TD
Req["request"] --> Key["compute key"]
Key --> Has{"in cache?"}
Has -->|"yes"| Ret["return stored value"]
Has -->|"no"| Comp["compute answer"]
Comp --> Store["store under key"]
Store --> Ret
style Ret fill:#166534,color:#fff
style Comp fill:#b45309,color:#fff
Measuring it: the hit rate
A cache that never hits is dead weight — extra code, extra memory, zero savings. So the first thing to measure is the hit rate: the share of lookups that found what they wanted. It is hits divided by total lookups (hits plus misses). A workload of ten requests where three were repeats gives three hits out of ten, a rate of 0.3.
The hit rate tells you whether caching is even working before you trust any savings number. A rate near zero means your keys are wrong (the usual culprit, coming up) or your workload genuinely has no repetition. A rate above a half on a realistic workload usually means the cache is earning its keep. Watch the number, not your intuition about how repetitive things 'should' be — the cache will quietly report exactly how helpful it is.
function hitRate(hits, misses) {
const total = hits + misses;
return total === 0 ? 0 : hits / total;
}
console.log(hitRate(0, 0)); // no data yet
console.log(hitRate(3, 1)); // 3 of 4
console.log(hitRate(8, 2)); // 0.8
Key design: what makes two requests 'the same'
The cache only works if its key captures exactly the inputs that change the answer and ignores everything else. Put too much in the key and identical questions get different keys, so nothing hits. Put too little in and different questions share a key, so a request gets back somebody else's answer — a silent, dangerous bug that looks like the cache is working beautifully while serving the wrong results.
The safe ingredients are the parts that affect the model's output: the prompt itself, the model name, and the generation settings (temperature in particular). The things to leave out are everything per-call: a request id, a timestamp, a trace identifier. None of them change the answer, so none of them belong in the key.
flowchart TD P["prompt"] --> K["stable cache key"] M["model + settings"] --> K ID["request id<br/>timestamp<br/>trace id"] -.->|"exclude"| X["kept OUT of the key"] K --> S["same question -> same key"] style K fill:#166534,color:#fff style X fill:#b91c1c,color:#fff
flowchart LR W["workload<br/>a a b a b c"] --> Proc["6 lookups"] Proc --> H["3 hits<br/>3 misses"] H --> R["hit rate = 0.5"] style R fill:#166534,color:#fff
function makeCache() {
const store = new Map();
let hits = 0, misses = 0;
return {
get(key) {
if (store.has(key)) { hits++; return store.get(key); }
misses++;
return undefined;
},
set(key, value) { store.set(key, value); },
hitRate() {
const total = hits + misses;
return total === 0 ? 0 : hits / total;
}
};
}
function ask(question, compute, cache) {
const cached = cache.get(question);
if (cached !== undefined) return cached;
const answer = compute(question);
cache.set(question, answer);
return answer;
}
let calls = 0;
const compute = (q) => { calls++; return 'answer to: ' + q; };
const workload = ['a', 'a', 'b', 'a', 'b', 'c'];
const cache = makeCache();
workload.forEach(q => ask(q, compute, cache));
console.log('compute calls =', calls);
console.log('hit rate =', cache.hitRate());
What does this print? make runs only when a lookup misses. The first x misses, the second x hits, then y misses.
const cache = new Map();
function lookup(key, make) {
if (cache.has(key)) return cache.get(key);
const v = make();
cache.set(key, v);
return v;
}
let calls = 0;
const make = () => { calls++; return calls * 10; };
lookup('x', make);
lookup('x', make);
lookup('y', make);
console.log(calls);
lookup('x') misses, so make runs once: calls becomes 1.
lookup('x') again is a hit, so make does not run. lookup('y') misses, so make runs again: calls becomes 2.
Write hitRate(hits, misses) that returns the fraction of lookups that were hits: hits / (hits + misses). If there were no lookups at all (hits + misses is 0), return 0.
function hitRate(hits, misses) {
// hits / (hits + misses); return 0 when there is no data
}
Total lookups is hits + misses.
Divide hits by the total — but if the total is 0, return 0 to avoid dividing by zero.
function hitRate(hits, misses) {
const total = hits + misses;
return total === 0 ? 0 : hits / total;
}
Write makeCache() returning an object with get(key), set(key, value), and hitRate(). Back it with a Map. get returns the stored value on a hit and undefined on a miss, and it must count each. hitRate() returns hits over total lookups, or 0 when there are none.
function makeCache() {
// return { get, set, hitRate } backed by a Map, counting hits and misses
}
Keep a Map for storage and two counters, hits and misses, closed over by the returned methods.
In get, branch on store.has(key): increment hits and return the value, else increment misses and return undefined.
function makeCache() {
const store = new Map();
let hits = 0, misses = 0;
return {
get(key) {
if (store.has(key)) { hits++; return store.get(key); }
misses++;
return undefined;
},
set(key, value) { store.set(key, value); },
hitRate() {
const total = hits + misses;
return total === 0 ? 0 : hits / total;
}
};
}
This cacheKey(req) should produce a STABLE key — the same logical question should always map to the same key. But it serialises the whole request, including a per-call id, so identical questions get different keys and nothing ever hits. Build the key from only the parts that affect the answer (the prompt and the model), not the request id.
function cacheKey(req) {
return JSON.stringify(req); // BUG: includes the per-request id, so nothing hits
}
JSON.stringify includes the id, so two requests with the same prompt but different ids get different keys.
Concatenate only req.prompt and req.model (with a separator), so the id no longer matters.
function cacheKey(req) {
return req.prompt + '|' + req.model;
}
Write cachedCall(prompt, compute, cache) where cache is a Map. If the prompt is already a key in the cache, return the stored value WITHOUT calling compute. Otherwise call compute(prompt), store the result under the prompt, and return it.
function cachedCall(prompt, compute, cache) {
// hit: return the stored value; miss: compute, store, return
}
Check cache.has(prompt) first; on a hit, return cache.get(prompt) and do not call compute.
On a miss, call compute(prompt), cache.set(prompt, result), and return result.
function cachedCall(prompt, compute, cache) {
if (cache.has(prompt)) return cache.get(prompt);
const result = compute(prompt);
cache.set(prompt, result);
return result;
}
Recap
- A cache stores answers by key; a hit returns the stored value and skips the work, a miss computes, stores, then returns.
- Every hit is a model call you did not make — tokens and latency both saved.
- The hit rate (hits / total lookups) is the first number to watch; near zero means the cache is not helping.
- A good key holds what changes the answer (prompt, model, settings) and excludes what doesn't (request id, timestamp, trace id).
- The classic trap: a key built from the whole request includes a unique id, so every lookup misses and the hit rate sits at zero.
Next you will right-size each call with a router — sending the easy ones to a cheaper model — which stacks on top of a cache for compounding savings.