Every call to a model is billed in tokens, and a token is roughly four characters of text. A request carries two token bills: the input (your prompt — system instructions, retrieved documents, conversation history, the user's question) and the output (the model's reply). For most real workloads the input dwarfs the output, because a short question still rides on top of a large, reused context.
That detail is the whole story of this lesson. Two requests that look different to a user — summarise doc A and summarise doc B — carry almost identical input: the same instructions, the same formatting rules, the same worked examples. Charged separately, you pay for that shared context twice. Batched into one call, you pay once. Truncation and compression attack the same problem from the other side, shrinking the context each call carries. You build all three and measure the reduction — no model required, just the arithmetic a billing page uses.
flowchart LR A["one request"] --> B["input: prompt tokens"] A --> C["output: completion tokens"] B --> D["reused every call<br/>the cost driver"] C --> E["small by comparison"] style D fill:#b45309,color:#fff style E fill:#166534,color:#fff
Counting tokens
Before you can optimise a bill you have to read it. Production tokenizers split text into sub-word units, but the arithmetic of cost is identical with any reasonable counter, so we use a word counter: a string's token count is the number of whitespace-separated runs it holds. Given a counter tok, the input cost of one call is tok(systemPrompt + query) and the output cost is tok(reply).
The structural fact to internalise: whatever sits in the system prompt is charged as input on every call. A 200-token prompt sent 100 times is 20 000 input tokens before a single user question is added. The completion, by contrast, is fresh each time and usually short. When a team says a feature is too expensive to ship, the fix nine times in ten is not a cheaper model; it is paying for the shared context fewer times.
function tok(s) {
return (s.match(/\S+/g) || []).length;
}
// Each separate call sends the system prompt again.
function separateInput(systemPrompt, queries) {
let total = 0;
for (const q of queries) total += tok(systemPrompt + " " + q);
return total;
}
console.log(tok("summarise this document"));
console.log(separateInput("you are a helper", ["a", "b", "c"]));
Why batching wins
Stack the items you were going to ask about into one request — one system prompt, then item 1, item 2, item 3 — and request all the answers at once. The shared context travels exactly once, so the input bill drops by the prompt size multiplied by how many calls you removed.
For N items each needing the system prompt S, N separate calls spend N * S tokens on shared context alone. One batched call spends S. The saving is (N - 1) * S input tokens, and it grows with how many items you can sensibly group. The catch is real: batching only helps when the items are independent and the combined reply stays short enough to stay reliable. Group a thousand unrelated questions and the model loses track long before you save anything. Batch the work that is genuinely shared.
flowchart TD S["system prompt S"] --> N["N separate calls<br/>pay S exactly N times"] S --> B["1 batched call<br/>pays S exactly once"] N --> R["saving: (N - 1) x S input tokens"] B --> R style R fill:#166534,color:#fff
function tok(s) {
return (s.match(/\S+/g) || []).length;
}
function batchedInput(systemPrompt, queries) {
return tok(systemPrompt + " " + queries.join(" "));
}
function separateInput(systemPrompt, queries) {
let total = 0;
for (const q of queries) total += tok(systemPrompt + " " + q);
return total;
}
const S = "you are a helpful assistant be concise";
const qs = ["translate hi", "translate bye", "translate thanks"];
console.log("separate:", separateInput(S, qs));
console.log("batched: ", batchedInput(S, qs));
console.log("saved: ", separateInput(S, qs) - batchedInput(S, qs));
From tokens to a bill
Tokens become money through a per-token rate, and the rate on output is usually several times the rate on input — a model spends more to generate a word than to read one. That asymmetry is why output length matters even when it looks small: a verbose reply billed at the output rate can quietly outweigh the input you worked so hard to trim. The same counter measures both, so once tok exists your cost is tok(prompt) * inputRate + tok(reply) * outputRate. Keep that formula in view for the scorecard lesson, where you plug real rates in and report a single cost-per-request number. For now, treat every token as money and the three techniques — batch, truncate, compress — as three ways to spend less of it.
Shrinking what each call carries
Batching pays for shared context fewer times; truncation and compression make the context itself smaller. Truncation is the blunt instrument: keep the system prompt and the most recent K turns, drop everything older. A chat that has run for fifty turns rarely needs turn one verbatim — the last three turns carry the working state. Compression is the lossy instrument: replace a run of raw items with a short summary of them, then carry the summary forward instead of the originals.
Both are bounded by a token budget — the most input tokens a single call may spend. Truncation is trivial to get right and lossless for whatever it keeps; compression reclaims more space but can drop a detail that later matters. The honest way to choose is to measure: set a budget, run your real workload, and read how many tokens each strategy removes.
flowchart LR H["full history<br/>10 turns"] --> T["truncate<br/>keep last K"] H --> P["compress<br/>summarise old turns"] T --> B["fits token budget"] P --> B style B fill:#166534,color:#fff
function tok(s) {
return (s.match(/\S+/g) || []).length;
}
function truncate(turns, budget) {
let kept = [];
let used = 0;
for (let i = turns.length - 1; i >= 0; i--) {
const cost = tok(turns[i]);
if (used + cost > budget) break;
used += cost;
kept.unshift(turns[i]);
}
return kept;
}
console.log(JSON.stringify(truncate(["old turn one two", "mid turn", "recent reply"], 4)));
What does this print? tok counts whitespace-separated runs.
function tok(s) {
return (s.match(/\S+/g) || []).length;
}
console.log(tok("hello world"));
console.log(tok("one two three four"));
Count the runs separated by spaces.
Two runs in the first, four in the second.
Write savings(systemPrompt, queries) that returns how many INPUT tokens batching saves versus making one separate call per query. tok is provided. Batched input is one call with the system prompt and all queries; separate input is one call per query, each carrying the system prompt.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function savings(systemPrompt, queries) {
// separate input minus batched input
}
Separate input sums tok(systemPrompt + q) over every query.
Batched input is tok(systemPrompt + all queries joined).
Return the difference; for one query the saving is 0.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function savings(systemPrompt, queries) {
let separate = 0;
for (const q of queries) separate += tok(systemPrompt + " " + q);
const batched = tok(systemPrompt + " " + queries.join(" "));
return separate - batched;
}
This totalInput is meant to count the input tokens for N separate calls, but it charges the system prompt ONCE instead of once per call. Three queries on a 4-token prompt should total 15, not 7. Fix it so the system prompt is counted on every call.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function totalInput(systemPrompt, queries) {
return tok(systemPrompt) + queries.reduce((a, q) => a + tok(q), 0);
}
The system prompt is sent on every call, so its tokens are multiplied by the number of queries.
tok(systemPrompt) needs to be multiplied by queries.length, not added once.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function totalInput(systemPrompt, queries) {
return tok(systemPrompt) * queries.length + queries.reduce((a, q) => a + tok(q), 0);
}
You have 40 independent product reviews to classify and a 150-token system prompt. Which change removes the MOST input tokens?
Batching pays the shared 150-token prompt once instead of 40 times, removing 39 * 150 = 5 850 input tokens. Option 4 is not real — each request must carry its own prompt — and the other two shave a handful at most.
Write truncate(turns, budget) that keeps the most RECENT turns (from the end of the array) until adding the next-older turn would exceed budget tokens, then returns the kept turns in their original chronological order. Each turn costs tok(turn) tokens. If even the newest turn exceeds the budget, return an empty array.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function truncate(turns, budget) {
// keep recent turns from the end until the budget would break
}
Walk from the last turn backward, tracking how many tokens you have spent.
Stop as soon as the next older turn would push the total over budget.
unshift each kept turn so the result stays in chronological order.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function truncate(turns, budget) {
let kept = [];
let used = 0;
for (let i = turns.length - 1; i >= 0; i--) {
const cost = tok(turns[i]);
if (used + cost > budget) break;
used += cost;
kept.unshift(turns[i]);
}
return kept;
}
Recap
- A request bills input tokens (prompt: system plus context plus query) and output tokens (reply); input usually dominates because the context is reused.
- A counter (
tok) turns text into a token bill, which is the first thing you need before you can shrink it. - Batching
Nindependent items into one call pays the shared system prompt once instead ofNtimes — a saving of(N - 1) * Sinput tokens. - Truncation keeps the system prompt plus the most recent
Kturns within a token budget; lossless for what it keeps. - Compression replaces raw items with a summary, reclaiming more space at the risk of losing a detail.
- The billing bug to fear: charging the system prompt once instead of per call, which erases batching's measured win.
Next you will turn these per-call savings into a scorecard that reports cost before and after, so the improvement is a number a reviewer can sign off on.