Proving the Savings

A feature that feels cheaper after optimisation is not a shipped feature — it is a rumour. The reviewer holding the budget, the finance team approving the model spend, and the on-call engineer watching a dashboard all need the same thing: a number. Specifically they need the cost of a request before the change and after it, in the same units, on the same workload, so the difference is a real reduction and not an artefact of how you counted.

This lesson builds that number. You already have the ingredients from the caching, routing, and batching lessons: a tokenizer, a per-token rate, and two ways to run a workload. Here you assemble them into a scorecard — a small structured object that reports cost-per-request before and after, the absolute saving, and the percentage cut. No model is called; every figure is arithmetic over a fixed fixture, which is exactly why a reviewer can trust it.

flowchart LR
  P["prompt tokens"] --> C["request cost"]
  R["reply tokens"] --> C
  C --> O["tok(prompt) x rateIn<br/>+ tok(reply) x rateOut"]
  style O fill:#1d4ed8,color:#fff

One request's cost is prompt tokens times the input rate plus reply tokens times the output rate.

The cost of one request

A request costs tok(prompt) * rateIn + tok(reply) * rateOut, where rateIn and rateOut are the per-token prices for input and output. Output is usually priced higher than input, so a reply that doubles in length can cost more than the prompt that produced it. Hold the rates as plain constants — their exact values drift as providers update pricing, but the shape of the formula does not, and that shape is what a scorecard depends on.

The unit you choose matters more than the number. Work in credits or micros and the scorecard compares whole numbers; work in dollars and floating-point noise creeps into every division. The examples below use whole-number credits so the arithmetic stays legible. A scorecard only stays honest when both sides share one unit — mixing them is the first way it learns to lie.

A cost helper: input tokens times rateIn plus output tokens times rateOut. Press Run.

function tok(s) {
  return (s.match(/\S+/g) || []).length;
}
function cost(prompt, reply, rateIn, rateOut) {
  return tok(prompt) * rateIn + tok(reply) * rateOut;
}

console.log(cost("a b c", "x y", 3, 12));
console.log(cost("short prompt", "a rather long reply indeed", 3, 12));

Cost per request, not cost per workload

A workload is a list of requests. Its total cost is the sum of each request's cost, but the headline number teams watch is cost per request — the total divided by how many requests ran. Per-request is what survives a comparison: if the optimised path also changed how many requests you make (batching turns forty calls into one), then comparing the two totals is meaningless, because the totals describe workloads of different sizes. Per-request normalises that difference away.

The rule that keeps a scorecard honest is to compare like with like. Before-per-request against after-per-request, or before-total against after-total — never before-total against after-per-request, and never divide one side's total by the other side's request count. Match the denominators first; everything else is formatting.

flowchart TD
  T["total cost"] --> D["divide by request count"]
  D --> PR["cost per request"]
  PR --> B["before"]
  PR --> A["after"]
  B --> CMP["compare like with like"]
  A --> CMP
  style CMP fill:#166534,color:#fff

Normalise both sides to cost-per-request before comparing, so a change in request count cannot fake a saving.

Cost-per-request sums every request then divides by the count. Run it.

function tok(s) {
  return (s.match(/\S+/g) || []).length;
}
function perRequestCost(workload, rateIn, rateOut) {
  if (workload.length === 0) return 0;
  let total = 0;
  for (const req of workload) {
    total += tok(req.prompt) * rateIn + tok(req.reply) * rateOut;
  }
  return total / workload.length;
}

const before = [{ prompt: "a b c", reply: "x y" }, { prompt: "d e", reply: "z" }];
console.log(perRequestCost(before, 3, 12));

Assembling the scorecard

With per-request cost in hand, the scorecard is four fields bolted onto two runs of the same workload. Run the workload the old way to get beforePerRequest; run it the optimised way to get afterPerRequest; subtract for the absolute savedPerRequest; and divide the saving by the before value for percentSaved. That last division is where most scorecards go wrong, so it gets its own exercise: the base of the percentage is the BEFORE cost, because the question is how much of the original spend did we cut, not how many times cheaper. Report the percentage rounded to a whole number so it reads cleanly, but keep the raw per-request values unrounded internally — rounding before dividing lets small errors accumulate into a wrong headline.

Why every number is a fixture

These exercises call no model and read no clock. Each workload is a fixed list of prompts and replies, the rates are constants, and the cost falls straight out of arithmetic. That determinism is the point: a scorecard you can recompute from a fixture is one a reviewer can re-check, and a CI job can re-run. In production the prompts and replies come from real traffic and real model output, but the scorecard code is identical — swap the fixture for a log of yesterday's requests and the same four fields come out. Build it against fixtures first, where the right answer is knowable by hand, and the production numbers stay trustworthy because the arithmetic never changed.

flowchart LR
  BF["before per request"] --> M["minus after per request"]
  AF["after per request"] --> M
  M --> S["saving"]
  S --> PCT["divide by BEFORE x 100<br/>= percent saved"]
  style PCT fill:#b45309,color:#fff

percentSaved divides the saving by the BEFORE per-request cost, then scales by 100. The base is always the before value.

Two ways a scorecard lies

The first lie is the percentage base: writing (before - after) / after instead of / before. For a drop from 100 to 40 credits, the honest cut is 60 percent; the inverted formula reports 150 percent and makes a solid optimisation look absurd. The second lie is mismatched denominators: batching collapsed forty requests into one, so the after-total describes one request while the before-total describes forty. Comparing those totals directly proves a 97 percent saving that is partly just you did less work. Always reduce both sides to cost-per-request first, then subtract and divide. A scorecard that mixes totals and per-request numbers always flatters the optimisation — and the reviewer who catches it trusts the whole feature less, not more.

Exercise

What does this print? A request costs input tokens times rateIn plus output tokens times rateOut.

function tok(s) {
  return (s.match(/\S+/g) || []).length;
}
function cost(prompt, reply, rateIn, rateOut) {
  return tok(prompt) * rateIn + tok(reply) * rateOut;
}
console.log(cost("a b c", "x y", 3, 12));

Your prediction — what does this print?

Exercise

Write perRequestCost(workload, rateIn, rateOut) where each request is an object { prompt, reply }. Return the average request cost: sum tok(prompt) * rateIn + tok(reply) * rateOut over every request, then divide by the number of requests. Return 0 for an empty workload. tok is provided.

function tok(s) { return (s.match(/\S+/g) || []).length; }

function perRequestCost(workload, rateIn, rateOut) {
  // sum each request's cost, then divide by the count
}

function tok(s) { return (s.match(/\S+/g) || []).length; }

function perRequestCost(workload, rateIn, rateOut) {
  if (workload.length === 0) return 0;
  let total = 0;
  for (const req of workload) {
    total += tok(req.prompt) * rateIn + tok(req.reply) * rateOut;
  }
  return total / workload.length;
}

Exercise

This percentSaved should report how much of the BEFORE cost was cut, but it divides by the AFTER cost instead. A drop from 33 to 15 credits per request should read 55 percent, not 120. Fix the base of the percentage.

function tok(s) { return (s.match(/\S+/g) || []).length; }
function perRequestCost(workload, rateIn, rateOut) {
  if (workload.length === 0) return 0;
  let total = 0;
  for (const req of workload) total += tok(req.prompt) * rateIn + tok(req.reply) * rateOut;
  return total / workload.length;
}

function percentSaved(before, after, rateIn, rateOut) {
  const b = perRequestCost(before, rateIn, rateOut);
  const a = perRequestCost(after, rateIn, rateOut);
  if (b === 0) return 0;
  return Math.round((b - a) / a * 100);
}

function tok(s) { return (s.match(/\S+/g) || []).length; }
function perRequestCost(workload, rateIn, rateOut) {
  if (workload.length === 0) return 0;
  let total = 0;
  for (const req of workload) total += tok(req.prompt) * rateIn + tok(req.reply) * rateOut;
  return total / workload.length;
}

function percentSaved(before, after, rateIn, rateOut) {
  const b = perRequestCost(before, rateIn, rateOut);
  const a = perRequestCost(after, rateIn, rateOut);
  if (b === 0) return 0;
  return Math.round((b - a) / b * 100);
}

Exercise

Batching turned 40 separate requests into 1 batched call. To prove the saving honestly, your scorecard should compare:

The finished scorecard on a small before/after fixture. Run it.

function tok(s) { return (s.match(/\S+/g) || []).length; }
function perRequestCost(workload, rateIn, rateOut) {
  if (workload.length === 0) return 0;
  let total = 0;
  for (const req of workload) total += tok(req.prompt) * rateIn + tok(req.reply) * rateOut;
  return total / workload.length;
}
function scorecard(before, after, rateIn, rateOut) {
  const beforePerRequest = perRequestCost(before, rateIn, rateOut);
  const afterPerRequest = perRequestCost(after, rateIn, rateOut);
  const savedPerRequest = beforePerRequest - afterPerRequest;
  const percentSaved = beforePerRequest > 0 ? Math.round(savedPerRequest / beforePerRequest * 100) : 0;
  return { beforePerRequest, afterPerRequest, savedPerRequest, percentSaved };
}

const before = [{ prompt: "summarise this document fully", reply: "a short summary" }];
const after = [{ prompt: "summarise doc", reply: "summary" }];
console.log(scorecard(before, after, 3, 12));

Exercise

Write scorecard(before, after, rateIn, rateOut) that returns { beforePerRequest, afterPerRequest, savedPerRequest, percentSaved }. Use cost-per-request for both sides; savedPerRequest is the difference; percentSaved is the saving as a percentage of the BEFORE cost, rounded to a whole number (0 if before is 0). perRequestCost is provided.

function tok(s) { return (s.match(/\S+/g) || []).length; }
function perRequestCost(workload, rateIn, rateOut) {
  if (workload.length === 0) return 0;
  let total = 0;
  for (const req of workload) total += tok(req.prompt) * rateIn + tok(req.reply) * rateOut;
  return total / workload.length;
}

function scorecard(before, after, rateIn, rateOut) {
  // return the four-field report
}

function tok(s) { return (s.match(/\S+/g) || []).length; }
function perRequestCost(workload, rateIn, rateOut) {
  if (workload.length === 0) return 0;
  let total = 0;
  for (const req of workload) total += tok(req.prompt) * rateIn + tok(req.reply) * rateOut;
  return total / workload.length;
}

function scorecard(before, after, rateIn, rateOut) {
  const beforePerRequest = perRequestCost(before, rateIn, rateOut);
  const afterPerRequest = perRequestCost(after, rateIn, rateOut);
  const savedPerRequest = beforePerRequest - afterPerRequest;
  const percentSaved = beforePerRequest > 0 ? Math.round(savedPerRequest / beforePerRequest * 100) : 0;
  return { beforePerRequest, afterPerRequest, savedPerRequest, percentSaved };
}

Recap

A scorecard turns it feels cheaper into a number a reviewer can sign off on: before and after cost-per-request on the same workload.
One request costs tok(prompt) * rateIn + tok(reply) * rateOut; keep rates as constants, because the formula's shape is what matters as prices drift.
The headline is cost per request — total divided by request count — because it survives workloads of different sizes.
Compare like with like: per-request to per-request, or total to total, never a mix.
percentSaved = (before - after) / before * 100, rounded; the base is the before cost, because you are measuring how much of the original spend you cut.
Keep raw values unrounded internally and round only the reported percentage.

With caching, routing, batching, and a scorecard in hand, you can take a feature that was too expensive to ship and show — in numbers — that it no longer is.

The cost of one request

Cost per request, not cost per workload

Assembling the scorecard

Why every number is a fixture

Recap

Checkpoint quiz

Go deeper — technical resources