Zoom out from any shipping AI feature — a support assistant, a document summariser, a coding copilot — and the same shape appears. To the user it looks like one box: they type, the model answers. Behind that box is a pipeline, and almost every stage in it is ordinary software.
This is the lesson that reframes the job. The model is one stage in the pipeline — the part that turns a prompt into fluent text. Every other stage is deterministic code you write, version, and unit-test. Once you can see the pipeline, AI engineering stops feeling like prompt-tuning magic and starts looking like the systems work you already know how to do.
flowchart LR R["request"] --> RT["router"] RT --> RET["retriever"] RET --> M["model"] M --> G["guard"] G --> A["response"] style RT fill:#166534,color:#fff style RET fill:#166534,color:#fff style G fill:#166534,color:#fff style M fill:#6d28d9,color:#fff
The stages around the model
A request usually passes through five kinds of stage, and only one of them is the model. A router inspects the request and decides what kind it is — and whether a model is even needed, or whether a cheaper handler will do. A retriever gathers the context the model needs: it searches an index, ranks the hits, and hands the best passages over. A tool layer exposes the actions the model is allowed to take, each with a schema that validates arguments. A budgeter counts tokens and caps what one request may cost. A guard inspects the model's reply before it reaches the user — checking length, format, and safety — and replaces anything that fails.
Four of those five stages are deterministic. You write them, you test them, you ship them.
function route(text) {
const t = text.toLowerCase();
if (t.startsWith("hi") || t.startsWith("hello")) return "greeting";
if (t.endsWith("?")) return "question";
return "task";
}
function modelStub(prompt) {
return "MODEL: " + prompt; // deterministic stand-in for a real model call
}
console.log(route("what time is it?"));
console.log(modelStub("hello"));
Why this is good news
Because the surrounding stages are deterministic, they have the properties ordinary software has. You can unit-test a router by handing it ten inputs and asserting the route each one takes. You can test a guard by feeding it a deliberately broken reply and checking that it returns the fallback. You can test a budgeter with fixed token counts and know the answer to the token before you run it. None of that is possible for the model itself, which is exactly why the value of an AI engineer is concentrated in the stages you can test, not the one you cannot.
flowchart TD P["a shipped AI feature"] --> D["deterministic, unit-tested code<br/>router, retriever, guard, budget, eval"] P --> N["non-deterministic core<br/>the model call itself"] style D fill:#166534,color:#fff style N fill:#6d28d9,color:#fff
The guard is where reliability is won
If one stage earns its keep, it is the guard. A raw model reply can be empty, far too long, off-topic, or structurally broken — and if it ships straight to the user, the feature looks broken even when the model did ninety percent of its job. The guard turns 'the model usually replies well' into 'the user always sees something valid': it validates, it truncates or rejects, and it substitutes a safe fallback when validation fails. That conversion — from probabilistic output to a guaranteed-valid result — is the whole reason the deterministic shell exists.
function route(text) {
const t = text.toLowerCase();
if (t.startsWith("hi") || t.startsWith("hello")) return "greeting";
if (t.endsWith("?")) return "question";
return "task";
}
function modelStub(prompt) { return "MODEL: " + prompt; }
function guard(output, maxLen) {
if (typeof output !== "string" || output.length === 0) return "(no answer)";
if (output.length > maxLen) return output.slice(0, maxLen);
return output;
}
function handle(request) {
const kind = route(request);
if (kind === "greeting") return "Hi there!"; // no model call needed
return guard(modelStub(request), 40);
}
console.log(handle("hello there"));
console.log(handle("what time is it?"));
flowchart LR R["request"] --> M["model, no shell"] M --> U["raw output reaches the user<br/>empty, too long, or unsafe"] style M fill:#b91c1c,color:#fff style U fill:#b91c1c,color:#fff
function withinBudget(usedTokens, requestTokens, cap) {
return usedTokens + requestTokens <= cap; // would this request blow the budget?
}
console.log(withinBudget(800, 150, 1000)); // 950 <= 1000
console.log(withinBudget(800, 300, 1000)); // 1100 <= 1000
One stage is not like the others
Look back at the five stages. The router branches on the request. The retriever ranks vectors by a formula. The tool layer matches against declared schemas. The budgeter adds tokens. The guard compares against rules. All of them, given the same input, return the same output — every time. That is what deterministic means, and it is what makes them testable.
The model is the exception. Given the same prompt, it may return a different answer each call, because it samples from a probability distribution. You cannot unit-test your way around that — but you do not have to. In the next lesson you will wrap the non-deterministic core in a deterministic shell that retries, validates, and falls back, so the system as a whole is reliable even though one stage inside it is not.
What does this print? The router reads the start and the end of the request.
function route(text) {
const t = text.toLowerCase();
if (t.startsWith("hi") || t.startsWith("hello")) return "greeting";
if (t.endsWith("?")) return "question";
return "task";
}
console.log(route("what time is it?"));
Lowercase the text first, then test its ends.
It does not start with hi or hello, but it does end with a question mark.
Write route(text) that classifies a request: return "greeting" if it starts with hi or hello (case-insensitive), "question" if it ends with ?, otherwise "task". Check the greeting condition first.
function route(text) {
// classify the request into greeting, question, or task
}
Lowercase the text with toLowerCase before testing.
startsWith checks the front, endsWith checks the back; test greeting before the question mark.
function route(text) {
const t = text.toLowerCase();
if (t.startsWith("hi") || t.startsWith("hello")) return "greeting";
if (t.endsWith("?")) return "question";
return "task";
}
This guard is supposed to protect the user from a broken model reply — but it just returns the raw output, shipping whatever the model said, even when it is empty or far too long. Make it validate: return "(no answer)" for empty output, trim anything longer than maxLen, and otherwise pass the output through.
function guard(output, maxLen) {
return output;
}
If output is empty (or not a string), return the fallback.
If output is longer than maxLen, return output.slice(0, maxLen).
function guard(output, maxLen) {
if (typeof output !== "string" || output.length === 0) return "(no answer)";
if (output.length > maxLen) return output.slice(0, maxLen);
return output;
}
Write handle(request) that composes the pipeline. Route the request; if it is a "greeting", return the canned reply "Hi there!" and skip the model entirely. Otherwise send the request through modelStub, then through guard with a max length of 40, and return the result. route, modelStub, and guard are provided for you.
function route(text) {
const t = text.toLowerCase();
if (t.startsWith("hi") || t.startsWith("hello")) return "greeting";
if (t.endsWith("?")) return "question";
return "task";
}
function modelStub(prompt) { return "MODEL: " + prompt; }
function guard(output, maxLen) {
if (typeof output !== "string" || output.length === 0) return "(no answer)";
if (output.length > maxLen) return output.slice(0, maxLen);
return output;
}
function handle(request) {
// route; greet with no model; otherwise modelStub then guard(40)
}
Call route first; check for greeting before touching the model.
Otherwise return guard(modelStub(request), 40).
function route(text) {
const t = text.toLowerCase();
if (t.startsWith("hi") || t.startsWith("hello")) return "greeting";
if (t.endsWith("?")) return "question";
return "task";
}
function modelStub(prompt) { return "MODEL: " + prompt; }
function guard(output, maxLen) {
if (typeof output !== "string" || output.length === 0) return "(no answer)";
if (output.length > maxLen) return output.slice(0, maxLen);
return output;
}
function handle(request) {
const kind = route(request);
if (kind === "greeting") return "Hi there!";
return guard(modelStub(request), 40);
}
In a real AI feature, which of these stages is deterministic software you can unit-test?
The router is ordinary code that branches on the request, so it is deterministic and unit-testable. Generation, weights, and token probabilities are the non-deterministic core you do not own at runtime.
Recap
- A shipping AI feature is a pipeline, not one box: request → router → retriever → model → guard → response.
- Five stage kinds appear again and again: router, retriever, tool layer, budgeter, guard. Only the model is non-deterministic.
- Because the surrounding stages are deterministic, you can unit-test them the way you test any code — feed fixed inputs, assert fixed outputs.
- The guard converts 'the model usually replies well' into 'the user always sees something valid' by validating and substituting a fallback.
- The classic trap is shipping a model with no shell: raw output, good or bad, reaches the user.
Next you will build the deterministic shell itself — retries, validation, and fallback — that tames the one non-deterministic stage.