Your prototype ran beautifully in the playground: a clever prompt, a clean answer, a bill of roughly zero. Then you wired it to real traffic and the month-end invoice was a shock. What changed between it worked in the playground and it is affordable in production? The unit of account.
A model is not billed by the call, the word, or the character. It is billed by the token, and it bills in two directions. Every token you send is input; every token the model writes back is output. Those two streams are priced separately, and the output stream is the expensive one. Until you can compute a bill from those four numbers you are guessing, and guesses that feel small in the playground become large in production.
flowchart LR I["input tokens<br/>what you send"] --> B["one bill"] O["output tokens<br/>what comes back"] --> B B --> F["in x priceIn<br/>+ out x priceOut"] style B fill:#8b5cf6,color:#fff style F fill:#166534,color:#fff
How a single call is priced
Providers quote two numbers, both per million tokens: a price for input and a higher price for output. A call that sends 1,500 tokens and receives 300 costs 1500 * priceIn / 1_000_000 + 300 * priceOut / 1_000_000 dollars. That division by a million is where most off-by-a-thousand bugs are born, so keep it explicit in the code rather than in your head.
Output is pricier for a real reason. Reading input is a single pass over text the provider already has; generating output is autoregressive work, writing one token at a time, each step conditioned on everything before it. That is more compute per token, and the price reflects it. A short reply to a huge prompt can cost more than the prompt did, because the billing flows both ways and those ways do not flow equally.
function costUsd(inputTokens, outputTokens, priceIn, priceOut) {
// prices are per MILLION tokens, so divide by 1e6
return inputTokens * priceIn / 1e6 + outputTokens * priceOut / 1e6;
}
const input = 1500, output = 300, priceIn = 3, priceOut = 15;
console.log(`input cost: $${costUsd(input, 0, priceIn, priceOut).toFixed(4)}`);
console.log(`output cost: $${costUsd(0, output, priceIn, priceOut).toFixed(4)}`);
console.log(`total: $${costUsd(input, output, priceIn, priceOut).toFixed(4)}`);
Where the money actually hides
A single call's bill is small. The shock comes from volume you did not account for, and most of that volume is input you barely think about. The system prompt — your instructions, the persona, the output format — is sent on every single call, so a 2,000-token system prompt across a million calls is two billion input tokens before any user has typed anything. Retrieved documents inflate input further, and so does any text you paste in for context.
This is why teams who track #calls as their cost metric get blindsided. Two products with identical call counts can differ in bill by a thousandfold, because one sends a fifty-token prompt and the other sends a fifty-thousand-token one. The honest metric is tokens, not calls.
flowchart TD T1["turn 1 resends<br/>system + msg 1"] -.-> T2["turn 2 resends<br/>system + msg 1 + msg 2"] -.-> T3["turn 3 resends<br/>system + msg 1 + msg 2 + msg 3"] style T3 fill:#b91c1c,color:#fff
function inputBilledPerTurn(msgSizes, systemTokens) {
let running = 0;
const perTurn = [];
for (let i = 0; i < msgSizes.length; i++) {
running += msgSizes[i];
perTurn.push(systemTokens + running);
}
return perTurn;
}
const turns = [120, 120, 120, 120];
const system = 800;
const billed = inputBilledPerTurn(turns, system);
billed.forEach((t, i) => console.log(`turn ${i + 1}: ${t} input tokens`));
console.log(`total billed: ${billed.reduce((a, b) => a + b, 0)} input tokens`);
Read those numbers carefully
Four turns of a 120-token message on top of an 800-token system prompt billed 4,400 input tokens, not the 4 times 920 a naive multiply would suggest. The extra is pure re-transmission. Stretch that conversation to forty turns and the input billed grows roughly with the square of the length, because turn n pays for all n-1 turns before it.
This is why a chat product's cost curve bends upward: it is not the number of messages, it is that every message drags the entire history behind it. The same logic swallows the system prompt — it is resend number one on turn one and resend number one-thousand on turn one-thousand.
The lever that fixes it: caching
Because the same long prefix is sent over and over, providers let you cache it. On the first call you pay the normal input price to write the cache; on later calls that prefix is read back at a steep discount, often around a tenth of the fresh input rate. The system prompt and the fixed head of a conversation are ideal candidates: written once, read cheaply thousands of times.
The numbers in the demo below are illustrative — real rates live on each provider's pricing page and drift over time — but the shape is durable: a cached read is dramatically cheaper than a fresh one. Your job is to compute both and feel the size of the gap.
flowchart LR F["fresh prefix<br/>full input price"] --> C["cached bill"] H["cached prefix<br/>discounted read"] --> C C --> S["same prefix, far cheaper<br/>on every later call"] style H fill:#166534,color:#fff style F fill:#b45309,color:#fff
const PRICE_IN = 3; // fresh input, per million tokens
const PRICE_OUT = 15; // generated output, per million tokens
const CACHE_READ = 0.3; // cached prefix read back, per million tokens
function costWithCache(inputTokens, outputTokens, cachedInput) {
const fresh = inputTokens - cachedInput;
return fresh * PRICE_IN / 1e6
+ cachedInput * CACHE_READ / 1e6
+ outputTokens * PRICE_OUT / 1e6;
}
const inputTokens = 10000, outputTokens = 500, cached = 9000;
const withCache = costWithCache(inputTokens, outputTokens, cached);
const noCache = inputTokens * PRICE_IN / 1e6 + outputTokens * PRICE_OUT / 1e6;
console.log(`with caching: $${withCache.toFixed(4)}`);
console.log(`without caching: $${noCache.toFixed(4)}`);
What does this print? Prices are per million tokens, so the result is divided by 1e6. There are no output tokens in this call.
function costUsd(inputTokens, outputTokens, priceIn, priceOut) {
return inputTokens * priceIn / 1e6 + outputTokens * priceOut / 1e6;
}
console.log(costUsd(2000, 0, 5, 15));
2000 input tokens at $5 per million.
2000 * 5 = 10000, then divide by 1,000,000.
Write costUsd(inputTokens, outputTokens, priceIn, priceOut) that returns the dollar cost of a call, where the two prices are quoted per million tokens. Add the input and output contributions separately.
function costUsd(inputTokens, outputTokens, priceIn, priceOut) {
// prices are per million tokens
}
Multiply each token count by its own per-million price.
Divide each product by 1e6, then add the two parts.
function costUsd(inputTokens, outputTokens, priceIn, priceOut) {
return inputTokens * priceIn / 1e6 + outputTokens * priceOut / 1e6;
}
This costUsd forgets that prices are quoted per million tokens, so it is off by a factor of a million — a $3 call reports 3000000. Add the one divisor that turns per-million pricing into dollars. Input and output are already priced separately; only the unit is wrong.
function costUsd(inputTokens, outputTokens, priceIn, priceOut) {
return inputTokens * priceIn + outputTokens * priceOut;
}
Prices are per 1,000,000 tokens, not per token.
Divide each product by 1e6 before adding.
function costUsd(inputTokens, outputTokens, priceIn, priceOut) {
return inputTokens * priceIn / 1e6 + outputTokens * priceOut / 1e6;
}
In a multi-turn chat with NO caching, what is sent as INPUT to generate the reply to the 5th user message?
A model has no memory between turns, so to answer turn 5 it is sent the system prompt and the whole transcript up to and including message 5. That re-transmission is exactly why chat input costs climb faster than the message count.
Write totalInputBilled(msgSizes, systemTokens) returning the TOTAL input tokens billed across a whole conversation. msgSizes is the token size of each turn's new message. To answer turn i, the model is sent systemTokens plus every message from 0 through i. Sum that over all turns.
function totalInputBilled(msgSizes, systemTokens) {
// turn i resends systemTokens + msgSizes[0] + ... + msgSizes[i]
}
Keep a running sum of message sizes as the loop advances.
Each turn adds systemTokens plus that running sum to the grand total.
function totalInputBilled(msgSizes, systemTokens) {
let total = 0;
let running = 0;
for (let i = 0; i < msgSizes.length; i++) {
running += msgSizes[i];
total += systemTokens + running;
}
return total;
}
Recap
- A model bills in two directions: input tokens (what you send) and output tokens (what it writes), each at its own per-million price.
- Output is pricier because generating is autoregressive compute, not a single read over existing text.
- Single-call cost is
in * priceIn / 1e6 + out * priceOut / 1e6. The/ 1e6is the unit that bites — per-thousand vs per-million is a 1,000x mistake. - Spend hides in input volume: the system prompt repeated every call, retrieved docs, and the whole transcript resent each turn, which makes chat input grow roughly with the square of the length.
- Caching turns a repeated prefix from a full-price read into a cheap one, often around a tenth of the cost.
Next you will see why where you place information in that context matters as much as how much of it you can afford — the lost-in-the-middle effect.