Where the Money Actually Goes

An AI feature that dazzles in the demo can still die in production — not because the answers are wrong, but because the monthly bill arrives. The most useful lever you own is knowing where, exactly, the tokens are going. Optimising the wrong step feels productive and saves nothing, which is how a team spends a fortnight trimming the model's replies only to watch the invoice barely move.

This lesson is about attribution: labelling each stage of a request with the tokens it burns and the cost that follows, so the expensive step becomes a concrete number rather than a hunch. The obvious suspect is frequently innocent. You will build the whole accounting layer in plain JavaScript — no model calls, because cost is arithmetic you can run against a fixed fixture. Once spend is a sorted list, deciding what to cut stops being a guessing game.

The unit is the token, and inputs are not free

A model does not bill by the character or the word. It bills by the token, the chunk its tokenizer slices text into — roughly four characters of English per token, so a hundred-word paragraph is in the neighbourhood of a hundred and thirty tokens. Two streams make the bill, and they are priced separately. Input tokens are everything flowing in: the system prompt, the retrieved documents, the prior turns of conversation, the user's question. Output tokens are what the model generates back.

Output is almost always priced higher per token, because producing text is more costly work than merely reading it. So a step's cost is never one quantity times one rate. It is two quantities, each multiplied by its own rate, then added. Confuse the two and every number downstream turns into fiction.

flowchart LR
  S["system prompt<br/>1000 tok"] --> R["retrieved docs<br/>4000 tok"]
  R --> U["user message<br/>500 tok"]
  U --> M["model reply<br/>500 tok"]
  M --> C["cost attributed<br/>to each stage"]
  style C fill:#8b5cf6,color:#fff

A request is a pipeline of token-consuming stages; cost is attributed to each rather than lumped onto the model.

Costing a single step

Hand a stage the two numbers that matter and the cost writes itself. A step carries input_tokens and output_tokens. A price table carries input and output rates, each quoted per thousand tokens. The cost of the step is its inputs divided by a thousand times their rate, plus its outputs divided by a thousand times theirs — nothing more.

The rates in these examples are illustrative. Real per-token prices drift every quarter and differ across providers, so quoting a specific number here would be stale by the time you read it. But the shape is durable: two rates, a divide-by-a-thousand, an addition. Learn the shape and you can drop in any vendor's current price sheet without touching the logic. Separating the durable arithmetic from the volatile numbers is precisely what keeps cost code honest as the model catalogue churns.

A step's cost: inputs and outputs, each divided by 1000 and multiplied by its own rate. Press Run.

const price = { input: 2, output: 6 };  // units per 1k tokens (illustrative)

function stepCost(step, p) {
  return step.input_tokens / 1000 * p.input
       + step.output_tokens / 1000 * p.output;
}

console.log(stepCost({ input_tokens: 4000, output_tokens: 500 }, price));
console.log(stepCost({ input_tokens: 1000, output_tokens: 500 }, price));

flowchart TD
  I["input tokens"] -->|"inputs x input_rate"| Cost["step cost"]
  O["output tokens"] -->|"outputs x output_rate"| Cost
  Cost --> R["one number per stage"]
  style Cost fill:#166534,color:#fff

Inputs and outputs each take their own path to the cost — two rates, not one.

Attributing a whole pipeline

A request is several stages chained together, so the pipeline's cost is the sum of its steps' costs — and the interesting question is which step that sum is mostly made of. Walk the stages, cost each with the same function, add them for a total, and remember the largest as you go. That largest step is your real target: it is the one place a cut actually shifts the invoice.

This is where attribution earns its keep. Stare only at the total and you optimise blindly, almost always attacking the model's reply because that is the part people notice being produced. Summing per step turns a vague 'this request is expensive' into a ranked list, and the entry at the top of that list is frequently a surprise. Surprise is the whole point — it tells you the intuition that sent you after the wrong target was wrong.

Cost the whole pipeline, then name the step that dominates it. Run it.

const price = { input: 2, output: 6 };

function stepCost(step, p) {
  return step.input_tokens / 1000 * p.input + step.output_tokens / 1000 * p.output;
}
function totalCost(steps, p) {
  return steps.reduce((sum, s) => sum + stepCost(s, p), 0);
}
function mostExpensiveStep(steps, p) {
  let best = steps[0];
  for (const s of steps) if (stepCost(s, p) > stepCost(best, p)) best = s;
  return best.name;
}

const pipeline = [
  { name: "system prompt",  input_tokens: 1000, output_tokens: 0   },
  { name: "retrieved docs", input_tokens: 4000, output_tokens: 0   },
  { name: "user message",   input_tokens: 500,  output_tokens: 0   },
  { name: "model reply",    input_tokens: 0,    output_tokens: 500 }
];
console.log("total =", totalCost(pipeline, price));
console.log("biggest =", mostExpensiveStep(pipeline, price));

flowchart TD
  M["model reply<br/>500 tok x rate 6 = 3"] --> Who["which is biggest?"]
  R["retrieved docs<br/>4000 tok x rate 2 = 8"] --> Who
  Who --> W["retrieved docs win<br/>an input, not the output"]
  style W fill:#b45309,color:#fff

A cheaper-per-token input can still win when there are far more of them.

Rank every stage by cost; the top of the list is your optimisation target. Run it.

const price = { input: 2, output: 6 };
function stepCost(s, p) { return s.input_tokens / 1000 * p.input + s.output_tokens / 1000 * p.output; }

const pipeline = [
  { name: "system prompt",  input_tokens: 1000, output_tokens: 0   },
  { name: "retrieved docs", input_tokens: 4000, output_tokens: 0   },
  { name: "user message",   input_tokens: 500,  output_tokens: 0   },
  { name: "model reply",    input_tokens: 0,    output_tokens: 500 }
];

const ranked = pipeline
  .map(s => ({ name: s.name, cost: stepCost(s, price) }))
  .sort((a, b) => b.cost - a.cost);

ranked.forEach(r => console.log(r.name + ": " + r.cost));

Exercise

What does this print? A step's cost is its inputs over a thousand times their rate, plus its outputs over a thousand times theirs.

const price = { input: 2, output: 6 };
function stepCost(step, p) {
  return step.input_tokens / 1000 * p.input + step.output_tokens / 1000 * p.output;
}
console.log(stepCost({ input_tokens: 1000, output_tokens: 500 }, price));

Your prediction — what does this print?

Exercise

Write stepCost(step, p) that returns a step's cost: step.input_tokens / 1000 * p.input + step.output_tokens / 1000 * p.output.

function stepCost(step, p) {
  // inputs and outputs, each divided by 1000 and multiplied by its rate
}

function stepCost(step, p) {
  return step.input_tokens / 1000 * p.input + step.output_tokens / 1000 * p.output;
}

Exercise

Write mostExpensiveStep(steps, p) that returns the name of the step with the highest cost. stepCost is provided. Assume at least one step; on a tie, return the earlier one.

function stepCost(step, p) {
  return step.input_tokens / 1000 * p.input + step.output_tokens / 1000 * p.output;
}

function mostExpensiveStep(steps, p) {
  // return the name of the highest-cost step
}

function stepCost(step, p) {
  return step.input_tokens / 1000 * p.input + step.output_tokens / 1000 * p.output;
}

function mostExpensiveStep(steps, p) {
  let best = steps[0];
  for (const s of steps) {
    if (stepCost(s, p) > stepCost(best, p)) best = s;
  }
  return best.name;
}

Exercise

This stepCost should cost both inputs and outputs, but it counts only the output tokens — so a 4 000-token retrieved-context stage looks free. Add the missing input term. (A step with 4 000 input and 0 output should cost 8 at these rates, not 0.)

function stepCost(step, p) {
  return step.output_tokens / 1000 * p.output;   // BUG: ignores input_tokens entirely
}

function stepCost(step, p) {
  return step.input_tokens / 1000 * p.input + step.output_tokens / 1000 * p.output;
}

Exercise

Write dominantShare(steps, p) that returns the fraction (from 0 to 1) of the TOTAL pipeline cost spent on the single most expensive step. stepCost is provided; you handle the running total, the running maximum, and the division. If the total is 0, return 0.

function stepCost(step, p) {
  return step.input_tokens / 1000 * p.input + step.output_tokens / 1000 * p.output;
}

function dominantShare(steps, p) {
  // fraction of total cost on the biggest step; 0 if total is 0
}

function stepCost(step, p) {
  return step.input_tokens / 1000 * p.input + step.output_tokens / 1000 * p.output;
}

function dominantShare(steps, p) {
  let total = 0;
  let max = 0;
  for (const s of steps) {
    const c = stepCost(s, p);
    total += c;
    if (c > max) max = c;
  }
  return total === 0 ? 0 : max / total;
}

Recap

A request is a pipeline of token-consuming stages; the bill is the sum of their costs, not one number.
The unit is the token; input tokens (sent in) and output tokens (generated back) are priced at different rates, output usually higher.
Step cost = input_tokens / 1000 * input_rate + output_tokens / 1000 * output_rate. Two rates, a divide-by-a-thousand, an add.
Attribution means costing each stage, summing for a total, and tracking the largest — your real optimisation target.
The trap: assuming the reply dominates. Inputs like retrieved context are cheaper each but far more numerous, so they often top the list.

Next you will stop paying twice for the same answer by adding a cache — the highest-leverage cut there is, and a direct use of the ranked list you just learned to read.

Where the Money Actually Goes

The unit is the token, and inputs are not free

Costing a single step

Attributing a whole pipeline

Recap

Checkpoint quiz

Go deeper — technical resources