Model Routing: Right-Sizing Each Call

Not every question deserves the biggest model. A greeting, a one-word lookup, a templated formatting job — these are trivial, and routing them through a flagship model is like hiring a senior architect to change a lightbulb. The strong model handles them fine, but so would a model costing a tenth as much. The reverse is also true: send a hard reasoning task to the cheap model and you either get a wrong answer or a refund request.

Model routing is the practice of matching each request to the cheapest model that can still handle it. The payoff is direct and measurable: the same workload, the same answers, a smaller bill. This lesson builds a router driven by a cheap heuristic signal, then costs the workload before and after so the saving becomes a number you can put in a report — not a vibe.

A cheap signal for difficulty

The router must decide 'easy or hard?' before it spends any money, which rules out the obvious idea of asking a model to classify the request — that classification is itself a paid call, and it can easily cost more than it saves. Instead you want a signal you can compute for free, in milliseconds, from the request alone.

A workable first signal is a complexity score: start at the word count, then bump it up whenever the prompt contains a reasoning verb like explain, why, or compare. It is a blunt instrument — it will misclassify the occasional short-but-hard prompt — but it is cheap, transparent, and tunable. Compare the score to a threshold: at or above goes to the strong model, below goes to the weak one. The threshold is the dial you turn to trade savings against quality.

flowchart LR
  Req["request"] --> CX["complexity score"]
  CX --> Gate{"score high enough?"}
  Gate -->|"yes"| Strong["strong model<br/>handles hard tasks"]
  Gate -->|"no"| Weak["weak model<br/>cheap for easy tasks"]
  style Strong fill:#8b5cf6,color:#fff
  style Weak fill:#166534,color:#fff

A free heuristic scores each request; the threshold sends high scores to the strong model and low scores to the weak one.

A complexity score from word count plus keyword hits, then a threshold gate. Press Run.

function complexityOf(prompt) {
  const words = prompt.trim().split(/\s+/).filter(Boolean);
  let score = words.length;
  for (const w of ['explain', 'why', 'compare']) {
    if (prompt.toLowerCase().includes(w)) score += 5;
  }
  return score;
}
function route(prompt, threshold) {
  return complexityOf(prompt) >= threshold ? 'strong' : 'weak';
}

console.log(complexityOf('hi'), '->', route('hi', 5));
console.log(complexityOf('explain gravity'), '->', route('explain gravity', 5));

Costing both paths

To know whether routing helps, you cost the workload two ways and subtract. The first path sends every request to the strong model — the naive default, and your baseline. The second path routes each request and costs it on whichever model the router chose. The saving is the difference, and dividing by the baseline turns it into a fraction: a 30% reduction means thirty cents of every dollar stayed in your pocket.

To cost a call you need the tokens it would consume on each model. Real systems read those counts off the response; here we simulate them with a fixed fixture so the comparison is deterministic and costs nothing to run. The per-token rates are illustrative — the real ones drift every quarter — but the structure holds: pick a model, read its input and output rates, multiply by the simulated tokens, add.

flowchart LR
  All["all-strong cost<br/>the baseline"] --> Diff["difference = savings"]
  Routed["routed cost"] --> Diff
  Diff --> Pct["savings fraction"]
  style Diff fill:#166534,color:#fff
  style Pct fill:#8b5cf6,color:#fff

Savings = (all-strong cost - routed cost) / all-strong cost, a fraction of the baseline.

Cost the whole workload both ways and read the savings. Run it.

const MODELS = {
  weak:   { name: 'small', input_per_1k: 0.5, output_per_1k: 1.5 },
  strong: { name: 'large', input_per_1k: 3,   output_per_1k: 9   }
};
function complexityOf(p) {
  const words = p.trim().split(/\s+/).filter(Boolean);
  let s = words.length;
  for (const w of ['explain', 'why', 'compare']) if (p.toLowerCase().includes(w)) s += 5;
  return s;
}
function route(p, t) { return complexityOf(p) >= t ? 'strong' : 'weak'; }
function simulate(p, key) {
  const inTok = p.trim().split(/\s+/).filter(Boolean).length + 20;
  return { input_tokens: inTok, output_tokens: key === 'strong' ? 200 : 120 };
}
function costOf(p, key) {
  const s = simulate(p, key);
  return s.input_tokens / 1000 * MODELS[key].input_per_1k + s.output_tokens / 1000 * MODELS[key].output_per_1k;
}

const workload = ['hi', 'explain gravity', 'explain why the sky is blue'];
let allStrong = 0, routed = 0;
for (const p of workload) {
  allStrong += costOf(p, 'strong');
  routed += costOf(p, route(p, 5));
}
console.log('all strong =', allStrong.toFixed(2));
console.log('routed    =', routed.toFixed(2));
console.log('saved     =', ((allStrong - routed) / allStrong * 100).toFixed(0) + '%');

The threshold is a tradeoff

The threshold is not a number you discover, it is a dial you set — and both extremes are bad. Set it too low and almost everything scores above it, so nearly all traffic goes to the strong model and you save almost nothing. Set it too high and easy-looking but genuinely hard requests get dumped on the weak model, which either fails outright or returns a confidently wrong answer. The honest way to choose a threshold is to measure: run a labelled sample through the router at several thresholds, watching BOTH the savings and the quality (how many weak-model answers were actually correct), and settle on the point where you are still comfortable with the quality.

Routing is never perfectly accurate, and that is fine. The goal is a net win: enough correct cheap answers to matter, with the hard ones still reaching the model they need.

flowchart TD
  Low["threshold too low<br/>nearly all -> strong<br/>little saved"] --> Bad1["no savings"]
  Mid["threshold balanced<br/>easy -> weak, hard -> strong"] --> Good["savings + quality"]
  High["threshold too high<br/>hard -> weak<br/>quality drops"] --> Bad2["wrong answers"]
  style Good fill:#166534,color:#fff
  style Bad1 fill:#b45309,color:#fff
  style Bad2 fill:#b91c1c,color:#fff

Both threshold extremes lose: too low saves nothing, too high breaks quality. The dial sits between.

Don't pay to decide how to pay

The tempting design is to ask a model 'is this request hard?' and route on its answer. Resist it. The classifier is itself a model call, billed on every request, and it has to be right reliably enough to earn its keep — which a free heuristic usually does almost as well. A routing decision that costs money on every request eats the savings it was meant to create, and a router that depends on an LLM to classify difficulty has built a dependency that can fail, slow down, or price-spike independently of the work it routes.

Reach for the free signal first — word count, a keyword list, a regex, the presence of an attachment. Promote to a learned classifier only when measurement proves the heuristic is leaving real money on the table, and even then keep it on the cheapest model that classifies accurately.

Sweep the threshold and watch how much traffic goes to the weak model. Run it.

function complexityOf(p) {
  const words = p.trim().split(/\s+/).filter(Boolean);
  let s = words.length;
  for (const w of ['explain', 'why', 'compare']) if (p.toLowerCase().includes(w)) s += 5;
  return s;
}
function weakShare(workload, threshold) {
  let weak = 0;
  for (const p of workload) if (complexityOf(p) < threshold) weak++;
  return weak / workload.length;
}

const workload = ['hi', 'ok', 'summarise this', 'explain gravity', 'explain why', 'compare a and b', 'debug this error', 'translate this', 'greet the user', 'draft a poem'];
for (const t of [1, 5, 100]) {
  console.log('threshold', t, '-> weak share', weakShare(workload, t));
}

Exercise

What does this print? complexityOf starts at the word count and adds 5 for each reasoning keyword it finds.

function complexityOf(prompt) {
  const words = prompt.trim().split(/\s+/).filter(Boolean);
  let score = words.length;
  for (const w of ['explain', 'why', 'compare']) {
    if (prompt.toLowerCase().includes(w)) score += 5;
  }
  return score;
}
console.log(complexityOf('explain gravity'));

Your prediction — what does this print?

Exercise

Write route(prompt, threshold) that returns 'strong' when complexityOf(prompt) is at or above threshold, and 'weak' otherwise. complexityOf is provided.

function complexityOf(prompt) {
  const words = prompt.trim().split(/\s+/).filter(Boolean);
  let score = words.length;
  for (const w of ['explain', 'why', 'compare']) {
    if (prompt.toLowerCase().includes(w)) score += 5;
  }
  return score;
}

function route(prompt, threshold) {
  // 'strong' if complexity is at/above the threshold, else 'weak'
}

function complexityOf(prompt) {
  const words = prompt.trim().split(/\s+/).filter(Boolean);
  let score = words.length;
  for (const w of ['explain', 'why', 'compare']) {
    if (prompt.toLowerCase().includes(w)) score += 5;
  }
  return score;
}

function route(prompt, threshold) {
  return complexityOf(prompt) >= threshold ? 'strong' : 'weak';
}

Exercise

Write costOf(prompt, key, models) that returns the cost of one call to models[key], using the provided simulate. Cost is input_tokens / 1000 * models[key].input_per_1k + output_tokens / 1000 * models[key].output_per_1k.

const MODELS = {
  weak:   { name: 'small', input_per_1k: 0.5, output_per_1k: 1.5 },
  strong: { name: 'large', input_per_1k: 3,   output_per_1k: 9   }
};
function simulate(prompt, key) {
  const inTok = prompt.trim().split(/\s+/).filter(Boolean).length + 20;
  return { input_tokens: inTok, output_tokens: key === 'strong' ? 200 : 120 };
}

function costOf(prompt, key, models) {
  // simulate the call, then apply models[key] rates
}

const MODELS = {
  weak:   { name: 'small', input_per_1k: 0.5, output_per_1k: 1.5 },
  strong: { name: 'large', input_per_1k: 3,   output_per_1k: 9   }
};
function simulate(prompt, key) {
  const inTok = prompt.trim().split(/\s+/).filter(Boolean).length + 20;
  return { input_tokens: inTok, output_tokens: key === 'strong' ? 200 : 120 };
}

function costOf(prompt, key, models) {
  const s = simulate(prompt, key);
  return s.input_tokens / 1000 * models[key].input_per_1k + s.output_tokens / 1000 * models[key].output_per_1k;
}

Exercise

This route is inverted: it sends the HARD prompts to the weak model and the EASY ones to the strong model, which both wastes money and tanks quality. A high-complexity prompt should go to 'strong' and a low one to 'weak'. Fix the comparison. (complexityOf is provided.)

function complexityOf(prompt) {
  const words = prompt.trim().split(/\s+/).filter(Boolean);
  let score = words.length;
  for (const w of ['explain', 'why', 'compare']) {
    if (prompt.toLowerCase().includes(w)) score += 5;
  }
  return score;
}

function route(prompt, threshold) {
  return complexityOf(prompt) >= threshold ? 'weak' : 'strong';   // BUG: inverted
}

function complexityOf(prompt) {
  const words = prompt.trim().split(/\s+/).filter(Boolean);
  let score = words.length;
  for (const w of ['explain', 'why', 'compare']) {
    if (prompt.toLowerCase().includes(w)) score += 5;
  }
  return score;
}

function route(prompt, threshold) {
  return complexityOf(prompt) >= threshold ? 'strong' : 'weak';
}

Exercise

Write savings(workload, threshold, models) returning the cost reduction as a fraction (0 to 1): (allStrongCost - routedCost) / allStrongCost, where the all-strong cost sends every prompt to the strong model and the routed cost uses route per prompt. complexityOf, route, and simulate are provided. If the all-strong cost is 0, return 0.

function complexityOf(p) {
  const words = p.trim().split(/\s+/).filter(Boolean);
  let s = words.length;
  for (const w of ['explain', 'why', 'compare']) if (p.toLowerCase().includes(w)) s += 5;
  return s;
}
function route(p, t) { return complexityOf(p) >= t ? 'strong' : 'weak'; }
function simulate(p, key) {
  const inTok = p.trim().split(/\s+/).filter(Boolean).length + 20;
  return { input_tokens: inTok, output_tokens: key === 'strong' ? 200 : 120 };
}

function savings(workload, threshold, models) {
  // (allStrongCost - routedCost) / allStrongCost; 0 if allStrongCost is 0
}

function complexityOf(p) {
  const words = p.trim().split(/\s+/).filter(Boolean);
  let s = words.length;
  for (const w of ['explain', 'why', 'compare']) if (p.toLowerCase().includes(w)) s += 5;
  return s;
}
function route(p, t) { return complexityOf(p) >= t ? 'strong' : 'weak'; }
function simulate(p, key) {
  const inTok = p.trim().split(/\s+/).filter(Boolean).length + 20;
  return { input_tokens: inTok, output_tokens: key === 'strong' ? 200 : 120 };
}

function costAt(p, key, models) {
  const s = simulate(p, key);
  return s.input_tokens / 1000 * models[key].input_per_1k + s.output_tokens / 1000 * models[key].output_per_1k;
}
function savings(workload, threshold, models) {
  let all = 0, routed = 0;
  for (const p of workload) {
    all += costAt(p, 'strong', models);
    routed += costAt(p, route(p, threshold), models);
  }
  return all === 0 ? 0 : (all - routed) / all;
}

Recap

Routing matches each request to the cheapest model that can handle it; same workload, smaller bill.
The routing signal must be free and fast — a complexity score from word count plus keyword hits, not a paid model call.
Compare the score to a threshold: at or above goes to strong, below goes to weak.
Savings = (allStrongCost - routedCost) / allStrongCost, a fraction of the naive baseline.
The threshold is a tradeoff dial: too low saves nothing, too high breaks quality — pick it by measuring both.
The trap: paying a model to classify difficulty, which spends the savings it was meant to capture.

These three levers — attribute the spend, cache the repeats, route by difficulty — are the foundation of every token-economics decision that follows.

Model Routing: Right-Sizing Each Call

A cheap signal for difficulty

Costing both paths

The threshold is a tradeoff

Recap

Checkpoint quiz

Go deeper — technical resources