Call a language model twice with the same prompt and you may get two different answers. The model samples from a probability distribution, so its output is non-deterministic — and occasionally it is malformed, empty, or wrong. You cannot write a unit test that says 'the model replied correctly,' because correctness itself moves on every call.
Here is the turn that makes AI features shippable. You stop trying to test the model, and instead build a deterministic shell around it: code that retries, validates, and falls back. The model inside can be as flaky as it likes; the shell guarantees the system returns a correct, bounded result. You test the shell, not the core — and the shell is what you ship.
flowchart LR IN["a request"] --> S["deterministic shell<br/>retry + validate + fallback"] S --> C["the model<br/>(may return anything)"] C --> S S --> OUT["a guaranteed-valid result"] style S fill:#166534,color:#fff style C fill:#6d28d9,color:#fff
Three pieces, precisely named
To reason about this you need three words. The non-deterministic core is the model itself — given the same input, it may return anything. A stub (or fixture) is a deterministic stand-in for the model: same input, same output, always. You build stubs that pretend to be flaky — returning a bad reply for the first few calls — so you can test how the shell behaves without paying for or waiting on a real model. The deterministic shell is the code around the core: a retry loop, a validity check, and a fallback value. The shell is ordinary code; the stub lets you exercise it in a test; together they make the core's unpredictability manageable.
function makeFlakyModel(goodReply, failFirst) {
let calls = 0;
return function (prompt) {
calls += 1;
if (calls <= failFirst) return "ERR: malformed"; // simulate a bad model reply
return goodReply;
};
}
function safeCall(model, prompt, maxAttempts, fallback) {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
const reply = model(prompt);
if (reply !== "ERR: malformed") return reply; // got a valid reply
}
return fallback; // exhausted retries -> deterministic fallback
}
const model = makeFlakyModel("OK", 2); // fails its first 2 calls, then succeeds
console.log(safeCall(model, "hello", 3, "FALLBACK"));
How the shell earns its keep
The shell does three jobs in one loop. It calls the model. It checks whether the reply is valid — non-empty, well-formed, within bounds. If valid, it returns the reply; if not, it tries again, up to a fixed number of attempts. When the attempts run out, it returns a fallback you chose — a safe, deterministic value the feature can survive on. The result: no matter how often the core misbehaves, the shell either returns a good reply within the budget or the fallback. Both outcomes are ones you can test for, in advance, with a stub that fails on demand.
flowchart TD A1["attempt 1: malformed"] --> A2["attempt 2: malformed"] A2 --> A3["attempt 3: valid"] A3 --> R["return the valid reply"] A1 -.-> F["if all attempts fail,<br/>return the fallback"] style R fill:#166534,color:#fff style F fill:#b45309,color:#fff
The classic trap: call once, trust entirely
The trap is calling the model a single time and handing its reply straight to the user. Most of the time it works, which is what makes it seductive — until the model returns malformed JSON, an empty string, or a value out of range, and the feature crashes in production. The fix is to never treat a single model reply as an answer. Wrap every call in the shell, so that a bad reply triggers a retry and a run of bad replies lands on a fallback. The shell is a few lines of loop and a condition; the outage it prevents is the kind that loses users.
function makeFlakyModel(goodReply, failFirst) {
let calls = 0;
return function (prompt) {
calls += 1;
if (calls <= failFirst) return "ERR: malformed";
return goodReply;
};
}
function safeCall(model, prompt, maxAttempts, fallback) {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
const reply = model(prompt);
if (reply !== "ERR: malformed") return reply;
}
return fallback;
}
const alwaysBad = makeFlakyModel("OK", 9); // fails far more times than we will retry
console.log(safeCall(alwaysBad, "hello", 3, "try again later"));
Validity is a rule you choose
In the first example, 'valid' meant 'not the malformed sentinel.' That is the simplest rule, but the shell's real power is that validity is whatever you decide. For a JSON reply, valid might mean 'parses and has the required field.' For a classification, valid might mean 'the label is in the allowed set.' For a number, valid might mean 'greater than zero.' The shell does not care — it calls a predicate you pass in and retries until the predicate is satisfied or the attempts run out. One small machine, many definitions of 'good enough,' all of them testable.
function makeFlakyModel(goodReply, failFirst) {
let calls = 0;
return function (prompt) {
calls += 1;
if (calls <= failFirst) return "ERR: malformed";
return goodReply;
};
}
function callUntilValid(model, prompt, isValid, maxAttempts, fallback) {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
const reply = model(prompt);
if (isValid(reply)) return reply;
}
return fallback;
}
const model = makeFlakyModel("OK", 1);
const isNonEmpty = (r) => r !== "ERR: malformed";
console.log(callUntilValid(model, "hello", isNonEmpty, 3, "FALLBACK"));
flowchart LR R["request"] --> M["one raw model call,<br/>no shell"] M --> E["malformed reply<br/>feature crashes"] style M fill:#b91c1c,color:#fff style E fill:#b91c1c,color:#fff
What does this print? The stub fails its first call, then succeeds. The shell retries up to 3 times.
function makeFlakyModel(goodReply, failFirst) {
let calls = 0;
return function (prompt) {
calls += 1;
if (calls <= failFirst) return "ERR: malformed";
return goodReply;
};
}
function safeCall(model, prompt, maxAttempts, fallback) {
for (let a = 1; a <= maxAttempts; a++) {
const r = model(prompt);
if (r !== "ERR: malformed") return r;
}
return fallback;
}
const m = makeFlakyModel("OK", 1);
console.log(safeCall(m, "hi", 3, "FALLBACK"));
Call 1 returns malformed, so the shell tries again.
Call 2 returns OK, which is valid, so the shell returns it.
Write safeCall(model, prompt, maxAttempts, fallback) — the deterministic shell. Call model(prompt) up to maxAttempts times; return the first reply that is not "ERR: malformed". If every attempt is malformed, return fallback.
function safeCall(model, prompt, maxAttempts, fallback) {
// retry until a reply is valid, else return the fallback
}
Loop attempt from 1 to maxAttempts, calling model each time.
Return the first reply that is not malformed; after the loop, return fallback.
function safeCall(model, prompt, maxAttempts, fallback) {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
const reply = model(prompt);
if (reply !== "ERR: malformed") return reply;
}
return fallback;
}
This safeCall skips the shell entirely: it returns the very first model reply with no validation, no retry, and no fallback — so a malformed reply ships straight through. Make it retry up to maxAttempts times and fall back when all attempts are bad.
function makeFlakyModel(goodReply, failFirst) {
let calls = 0;
return function (prompt) {
calls += 1;
if (calls <= failFirst) return "ERR: malformed";
return goodReply;
};
}
function safeCall(model, prompt, maxAttempts, fallback) {
return model(prompt);
}
Wrap the call in a loop from 1 to maxAttempts.
Return the first non-malformed reply; after the loop, return fallback.
function makeFlakyModel(goodReply, failFirst) {
let calls = 0;
return function (prompt) {
calls += 1;
if (calls <= failFirst) return "ERR: malformed";
return goodReply;
};
}
function safeCall(model, prompt, maxAttempts, fallback) {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
const reply = model(prompt);
if (reply !== "ERR: malformed") return reply;
}
return fallback;
}
Write callUntilValid(model, prompt, isValid, maxAttempts, fallback) — the generalised shell. It calls model(prompt) up to maxAttempts times and returns the first reply for which isValid(reply) is true; if none pass, it returns fallback. makeFlakyModel is provided, but your shell must also work with any model and any validity rule.
function makeFlakyModel(goodReply, failFirst) {
let calls = 0;
return function (prompt) {
calls += 1;
if (calls <= failFirst) return "ERR: malformed";
return goodReply;
};
}
function callUntilValid(model, prompt, isValid, maxAttempts, fallback) {
// retry until isValid(reply) is true, else return the fallback
}
Loop, calling model each time, and test the reply with isValid.
The third test uses a numeric model and a predicate r > 10, so do not hardcode the malformed check.
function makeFlakyModel(goodReply, failFirst) {
let calls = 0;
return function (prompt) {
calls += 1;
if (calls <= failFirst) return "ERR: malformed";
return goodReply;
};
}
function callUntilValid(model, prompt, isValid, maxAttempts, fallback) {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
const reply = model(prompt);
if (isValid(reply)) return reply;
}
return fallback;
}
Why do you test the shell around a model instead of testing the model itself?
A model may return a different answer each call, so you cannot assert on its output. The shell is deterministic code — retry, validate, fallback — so you can point a flaky stub at it and prove it always returns a valid result or the fallback.
Recap
- A raw model call is non-deterministic: the same prompt can yield different, sometimes malformed, replies, so you cannot unit-test the model's output.
- A stub is a deterministic test double that fakes a flaky model, letting you exercise the shell without a real model.
- The deterministic shell calls the model, checks validity, retries up to a budget, and returns a fallback when the budget runs out — so the system is reliable even though its core is not.
- The classic trap is calling once and trusting entirely: one bad reply crashes the feature.
- Validity is a rule you choose; pass it in as a predicate and one shell serves every definition of 'good enough.'
This shell is the pattern the rest of the course builds on — retrieval, routing, and budgeting are all deterministic stages that make the model's output safe to ship.