From parts to a platform
Across this course you have built the parts of an AI system in isolation: a tokenizer that counts cost, a retriever that ranks context, a tool router that dispatches calls, a loop that feeds results back, a budgeter that caps the spend, and an eval harness that scores behaviour. Each was a small, pure function you could test on its own. A platform is what you get when those parts stop being islands and start passing one request through each other, in order, end to end.
This capstone wires them into a single runAgent(config, query) and grades the result. No model is called: the model is a stub returning scripted replies, so the whole assembly is verified with deterministic fixtures, the way a real eval suite grades a pipeline on every change.
flowchart LR TOK["tokenizer"] --> LOOP["dispatch loop<br/>+ step budget"] ROUTER["tool router"] --> LOOP LOOP --> RES["result object"] RES --> EVAL["eval harness<br/>vs golden traces"] style EVAL fill:#166534,color:#fff
The shape of runAgent
runAgent(config, query) returns one object, { answer, steps, tokens }, that summarises what happened. tokens is the input cost of the query, counted by the tokenizer. steps is how many model turns the loop took. answer is the content of the final assistant message. The config bundles everything the platform needs: a tools registry, a model, a tokenLimit the query must fit under, and a maxSteps budget that bounds the loop. Holding these in one config object is what makes the platform reproducible: the same config plus the same query always yields the same result, because nothing in the pipeline reads a clock or rolls a die.
flowchart TD
Q["query: add 2 and 3"] --> T["tokens: 4"]
T --> L["loop: 2 steps"]
L --> A["answer: 5"]
A --> OUT["{answer, steps, tokens}"]
style OUT fill:#1d4ed8,color:#fff
function tok(s) { return (s.match(/\S+/g) || []).length; }
function withinBudget(query, limit) { return tok(query) <= limit; }
function routeTool(tools, call) {
const fn = tools[call.name];
return fn ? { ok: true, result: fn(call.arguments) } : { ok: false, error: 'unknown tool' };
}
console.log(withinBudget('hello world foo', 5));
console.log(routeTool({ add: ({a,b}) => a + b }, { name: 'add', arguments: { a: 2, b: 3 } }).result);
console.log(routeTool({ add: () => 0 }, { name: 'missing', arguments: {} }).ok);
The router speaks one shape
A tool call can succeed or fail, and the rest of the platform should not have to tell those apart by sniffing types. So the router returns one normalized shape on every path: { ok: true, result } when the tool ran, or { ok: false, error } when the name was unknown or the function threw. That single ok flag is what lets the loop, the trace, and the eval all treat a tool call uniformly. Returning the raw result on success and a separate error object on failure (two different shapes) is the mistake that forces every downstream reader to special-case both.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function scriptedModel(scripts) { let i = 0; return { respond() { return scripts[i++]; } }; }
function runLoop(model, tools, messages, maxSteps) {
let steps = 0;
while (steps < maxSteps) {
const reply = model.respond(messages);
messages.push(reply);
steps++;
if (!reply.tool_calls || reply.tool_calls.length === 0) break;
for (const call of reply.tool_calls) {
const fn = tools[call.name];
const result = fn ? fn(call.arguments) : { error: 'unknown tool' };
messages.push({ role: 'tool', tool_call_id: call.id, content: JSON.stringify(result) });
}
}
return messages;
}
function runAgent(config, query) {
const messages = [{ role: 'user', content: query }];
const tokens = tok(query);
runLoop(config.model, config.tools, messages, config.maxSteps);
const assistants = messages.filter(m => m.role === 'assistant');
const answer = assistants.length ? (assistants[assistants.length - 1].content || '') : '';
return { answer, steps: assistants.length, tokens };
}
const config = {
tools: { add: ({a,b}) => a + b },
model: scriptedModel([
{ role: 'assistant', tool_calls: [{ id: 'c1', name: 'add', arguments: { a: 40, b: 2 } }] },
{ role: 'assistant', content: '42' }
]),
maxSteps: 5
};
console.log(runAgent(config, 'what is 40 plus 2').answer);
flowchart LR
G["golden traces"] --> CMP["compare each run"]
R["actual runs"] --> CMP
CMP --> S["{passed, total}"]
style S fill:#b45309,color:#fff
Golden traces grade behavior
An eval harness needs expected answers to compare against: golden traces, hand-written for a handful of representative queries. For each query you record what the correct final answer is, run the platform, and score the match. The score is not a vibe; it is answer === expected over every case, aggregated into { passed, total }. A change that lifts that ratio shipped an improvement; one that drops it regressed behavior, even if every unit test still passes. Because comparison is the only judgment call, normalize before you compare: trim whitespace and fold case, so a model that answers '42 ' is not marked wrong against a golden of '42'.
function scoreRun(result, expected) {
return { passed: String(result.answer).trim() === String(expected).trim() };
}
function evaluate(runs, goldens) {
let passed = 0;
for (let i = 0; i < goldens.length; i++) {
if (scoreRun(runs[i], goldens[i]).passed) passed++;
}
return { passed, total: goldens.length };
}
const runs = [{ answer: '42' }, { answer: ' 7 ' }, { answer: 'wrong' }];
const goldens = ['42', '7', '5'];
console.log(evaluate(runs, goldens));
Why the whole thing is gradable
Every part of this platform is pure over a fixture. The tokenizer is a whitespace split; the tools are plain functions; the router is a lookup; the model is a stub returning scripted replies in order; the budget is arithmetic. So runAgent is a pure function of its config and query: feed it the same two and you get the same { answer, steps, tokens } every time. That determinism is what lets a golden-trace eval grade it. Run a batch of queries, compare each answer to the expected one, and report { passed, total } without spending a cent. In production you swap the stub for a real model; the platform code does not change.
What does this print? tok splits on whitespace. The query has five words; the limit is four.
function tok(s) { return (s.match(/\S+/g) || []).length; }
const limit = 4;
const query = 'what is two plus two';
console.log(tok(query) + '/' + limit);
'what is two plus two' has five whitespace-separated tokens.
5 joined with '/' and the limit 4 gives '5/4'.
Write withinBudget(query, limit) that returns true when the query's token count is at or under the limit, and false otherwise. tok is provided.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function withinBudget(query, limit) {
// true if the query's token count is at or under the limit
}
Count tokens with tok(query).
Return whether that count is less than OR equal to the limit.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function withinBudget(query, limit) {
return tok(query) <= limit;
}
Write routeTool(tools, call) that looks up call.name in tools and runs it with call.arguments. Return { ok: true, result } on success, and { ok: false, error: 'unknown tool' } when the name is not in tools. One normalized shape on every path.
function routeTool(tools, call) {
// look up call.name; run it and return { ok: true, result },
// or { ok: false, error: 'unknown tool' } when the name is missing
}
Read tools[call.name]; if it is a function, run it and return { ok: true, result }.
Otherwise return { ok: false, error: 'unknown tool' }.
function routeTool(tools, call) {
const fn = tools[call.name];
if (fn) return { ok: true, result: fn(call.arguments) };
return { ok: false, error: 'unknown tool' };
}
This eval scores a run against a golden answer, but it compares with a raw ===, so a model answer that carries stray whitespace (' 5 ') fails a correct golden ('5'). Fix scoreRun to normalize both sides with trim() before comparing, so equivalent answers pass.
function scoreRun(result, expected) {
return { passed: result.answer === expected }; // BUG: stray whitespace fails a correct golden
}
Wrap both values in String(...) and call .trim() so surrounding whitespace is ignored.
The comparison stays ===, just on the trimmed strings.
function scoreRun(result, expected) {
return { passed: String(result.answer).trim() === String(expected).trim() };
}
Write runAgent(config, query) that assembles the platform. Tokenize the query for tokens, run the provided runLoop with config.model, config.tools, a fresh messages list (starting with the user query), and config.maxSteps, then return { answer, steps, tokens } where answer is the content of the last assistant message (or '') and steps is the count of assistant messages. tok, scriptedModel, and runLoop are already defined for you.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function scriptedModel(scripts) { let i = 0; return { respond() { return scripts[i++]; } }; }
function runLoop(model, tools, messages, maxSteps) {
let steps = 0;
while (steps < maxSteps) {
const reply = model.respond(messages);
messages.push(reply);
steps++;
if (!reply.tool_calls || reply.tool_calls.length === 0) break;
for (const call of reply.tool_calls) {
const fn = tools[call.name];
const result = fn ? fn(call.arguments) : { error: 'unknown tool' };
messages.push({ role: 'tool', tool_call_id: call.id, content: JSON.stringify(result) });
}
}
return messages;
}
function runAgent(config, query) {
// tokenise the query, run the loop, and return { answer, steps, tokens }
}
Start messages with the user query and compute tokens with tok(query).
Call runLoop(config.model, config.tools, messages, config.maxSteps) to fill the conversation.
Filter for role 'assistant' to get the step count; its last element's content is the answer.
function tok(s) { return (s.match(/\S+/g) || []).length; }
function scriptedModel(scripts) { let i = 0; return { respond() { return scripts[i++]; } }; }
function runLoop(model, tools, messages, maxSteps) {
let steps = 0;
while (steps < maxSteps) {
const reply = model.respond(messages);
messages.push(reply);
steps++;
if (!reply.tool_calls || reply.tool_calls.length === 0) break;
for (const call of reply.tool_calls) {
const fn = tools[call.name];
const result = fn ? fn(call.arguments) : { error: 'unknown tool' };
messages.push({ role: 'tool', tool_call_id: call.id, content: JSON.stringify(result) });
}
}
return messages;
}
function runAgent(config, query) {
const messages = [{ role: 'user', content: query }];
const tokens = tok(query);
runLoop(config.model, config.tools, messages, config.maxSteps);
const assistants = messages.filter(m => m.role === 'assistant');
const answer = assistants.length ? (assistants[assistants.length - 1].content || '') : '';
return { answer, steps: assistants.length, tokens };
}
Recap
- A platform is the course's parts (tokenizer, retriever, router, loop, budgeter, eval) wired so one request flows through them in order.
runAgent(config, query)returns{ answer, steps, tokens }: the input cost, the model-turn count, and the final assistant message.- The router normalizes every tool call to
{ ok, result | error }, so the loop and eval never branch on the shape of a value. - The budget bounds model turns (
maxSteps), not tokens. A turn is what costs money and time, so capping turns is what guarantees termination. - Because every part is pure over fixtures, the whole assembly is a pure function a golden-trace eval can grade on every change, with zero model calls.
You have now built a complete, deterministic AI system end to end: the infrastructure around a model, every piece of it testable without one.