An agent with a tool-dispatch loop looks robust, until one of its tools throws. A lookup service times out, a worker returns an error, a dependency is down for the minute. If a single failure aborts the whole run, the agent is fragile in the exact way production systems are not. Real systems expect failure and are designed around it: they retry the flaky call a bounded number of times, fall back to a safe alternative when retries run out, and stop hammering a dependency that is clearly dead.
This layer is deterministic and has nothing to do with the model. We stand in for a flaky dependency with a stub that fails its first few calls and then succeeds, and we assert on the recovery logic wrapped around it: how many attempts happened, whether the fallback fired, whether a broken circuit refused to call through. That logic is the whole lesson, and every assertion can be made without a live service.
flowchart TD
C["call the dependency"] --> OK{"succeeded?"}
OK -->|"yes"| R["return the result"]
OK -->|"no, attempts left"| C
OK -->|"no, none left"| F["use the fallback"]
style R fill:#166534,color:#fff
style F fill:#b45309,color:#fff
Bounded retry
The first defence is to try again. A call that fails once might be a transient blip, so callWithRetry calls the function up to maxAttempts times and returns the first value that comes back without throwing. The bound matters: without it, a call that fails forever retries forever, burning budget and stalling the agent. With it, the loop tries a fixed number of times and then gives up on purpose.
Give up how? Retries exist for transient failure, so when every attempt fails the right move is to surface the error, not hide it. The caller then decides whether to fall back or report the problem. Retry without a bound is a bug; retry without an honest failure path is a worse one, because it swallows the very signal the caller needed.
function callWithRetry(fn, maxAttempts) {
let lastErr;
for (let i = 0; i < maxAttempts; i++) {
try { return fn(); } catch (e) { lastErr = e; }
}
throw lastErr;
}
let calls = 0;
const flaky = () => {
calls++;
if (calls < 3) throw new Error('not yet');
return 'recovered on attempt ' + calls;
};
console.log(callWithRetry(flaky, 5));
console.log('attempts used: ' + calls);
Fallback: keep going without the dependency
Some calls are optional. If the recommendation service is down, the page can still render with a generic suggestion; if the live price feed fails, show the last cached price. A fallback is the cheaper or safer result you return when the primary call fails, so the agent keeps serving instead of crashing in front of the user.
The clean shape retries first, then falls back. A transient failure often clears up on the second or third try, so callWithRetryThenFallback tries the primary up to maxAttempts times, and only if every one of them throws does it return fallback(). The fallback is a deliberate degradation, not an accident, and it runs exactly when retries have been honestly exhausted rather than masked.
function callWithRetryThenFallback(fn, maxAttempts, fallback) {
let lastErr;
for (let i = 0; i < maxAttempts; i++) {
try { return fn(); } catch (e) { lastErr = e; }
}
return fallback();
}
let calls = 0;
const alwaysFails = () => { calls++; throw new Error('gone'); };
const result = callWithRetryThenFallback(alwaysFails, 3, () => 'cached value');
console.log(result);
console.log('attempts before fallback: ' + calls);
Circuit breaker: stop calling a dead dependency
Retries assume the failure is transient. When a dependency is genuinely down, retrying every request just piles load onto something already failing, a retry storm that turns a small outage into a big one. A circuit breaker watches consecutive failures, and once they cross a threshold it opens: subsequent calls fail immediately, without ever touching the dependency, until a cooldown passes.
The mechanics are small. The breaker holds a failure count and an open flag. A successful call resets the count to zero; a failed call increments it, and when it reaches the threshold the flag flips to open. While open, call throws without invoking the function at all. That is fail-fast: the agent stops waiting on a dead dependency and is free to use its fallback right away.
flowchart LR CLOSED["closed<br/>calls pass through"] -->|"failures reach threshold"| OPEN["open<br/>fail fast, skip fn"] OPEN -->|"cooldown passes"| HALF["half-open<br/>one trial call"] HALF -->|"trial succeeds"| CLOSED HALF -->|"trial fails"| OPEN style CLOSED fill:#166534,color:#fff style OPEN fill:#b91c1c,color:#fff style HALF fill:#b45309,color:#fff
function makeBreaker(threshold) {
let failures = 0;
let open = false;
return {
call(fn) {
if (open) throw new Error('circuit open');
try { const v = fn(); failures = 0; return v; }
catch (e) { failures++; if (failures >= threshold) open = true; throw e; }
},
isOpen() { return open; }
};
}
let calls = 0;
const dead = () => { calls++; throw new Error('down'); };
const breaker = makeBreaker(2);
for (let i = 0; i < 4; i++) {
try { breaker.call(dead); } catch (e) { /* counted as a failure */ }
}
console.log('fn called: ' + calls);
console.log('circuit open: ' + breaker.isOpen());
flowchart TD DOWN["dependency is down"] --> RETRY["keep retrying<br/>retry storm, wasted budget"] DOWN --> BREAK["breaker opens<br/>fail fast, use fallback"] style RETRY fill:#b91c1c,color:#fff style BREAK fill:#166534,color:#fff
What does this print? The stub fails on its first call and succeeds on its second. callWithRetry returns the first success and reports how many attempts it took.
function callWithRetry(fn, maxAttempts) {
let lastErr;
for (let i = 0; i < maxAttempts; i++) {
try { return fn(); } catch (e) { lastErr = e; }
}
throw lastErr;
}
let calls = 0;
const fn = () => { calls++; if (calls < 2) throw new Error('boom'); return 'ok'; };
const result = callWithRetry(fn, 3);
console.log(result);
console.log(calls);
First attempt: calls is 1, 1 < 2, so it throws.
Second attempt: calls is 2, 2 < 2 is false, so it returns 'ok'. Two calls were made.
Write callWithRetry(fn, maxAttempts) that calls fn up to maxAttempts times, returning the first value that does not throw. If every attempt throws, rethrow the last error. Count one attempt per call to fn.
function callWithRetry(fn, maxAttempts) {
// try fn up to maxAttempts times; return first success; else throw the last error
}
Loop with a try/catch: return on success, otherwise remember the error and keep going.
If the loop ends without returning, throw the last remembered error.
function callWithRetry(fn, maxAttempts) {
let lastErr;
for (let i = 0; i < maxAttempts; i++) {
try { return fn(); } catch (e) { lastErr = e; }
}
throw lastErr;
}
Write callWithRetryThenFallback(fn, maxAttempts, fallback) that retries fn up to maxAttempts times. If a call succeeds, return its value. If every attempt fails, return fallback() instead of throwing.
function callWithRetryThenFallback(fn, maxAttempts, fallback) {
// retry; if all attempts fail, return fallback()
}
Reuse the retry shape: try fn up to maxAttempts times, returning on success.
When the loop ends without success, return fallback() instead of throwing.
function callWithRetryThenFallback(fn, maxAttempts, fallback) {
let lastErr;
for (let i = 0; i < maxAttempts; i++) {
try { return fn(); } catch (e) { lastErr = e; }
}
return fallback();
}
This breaker increments its failure count and flips open after threshold failures, but it has no fail-fast check: once open it STILL calls fn on every subsequent call, hammering the dead dependency instead of refusing immediately. Add the missing check so that an open circuit throws new Error('circuit open') WITHOUT calling fn. With threshold 2 and four calls against an always-failing fn, fn should be invoked exactly twice.
function makeBreaker(threshold) {
let failures = 0;
let open = false;
return {
call(fn) {
try {
const v = fn();
failures = 0;
return v;
} catch (e) {
failures++;
if (failures >= threshold) open = true;
throw e;
}
},
isOpen() { return open; }
};
}
Add the guard at the very top of call, before fn is ever touched.
if (open) throw new Error('circuit open'); must run before the try block.
function makeBreaker(threshold) {
let failures = 0;
let open = false;
return {
call(fn) {
if (open) throw new Error('circuit open');
try {
const v = fn();
failures = 0;
return v;
} catch (e) {
failures++;
if (failures >= threshold) open = true;
throw e;
}
},
isOpen() { return open; }
};
}
Once a circuit breaker is open, why does it throw immediately instead of continuing to call the dependency?
Fail-fast prevents a retry storm: an open circuit refuses to call through, so the agent does not pile load onto a dead dependency and can degrade to its fallback without waiting for yet another timeout.
Recap
- Bounded retry calls a flaky function up to
maxAttemptstimes and returns the first success; when all attempts fail it throws the last error so the caller can react. - A fallback returns a safe, cheaper value when retries are exhausted, so the agent keeps serving instead of crashing.
- A circuit breaker counts consecutive failures and, past a threshold, opens: further calls fail fast without touching the dependency, stopping retry storms against a dead service.
- In production, pair retries with exponential backoff and jitter so retries spread out; the sandbox models that as an attempt count.
- Fail fast the moment the circuit opens, never keep calling a dependency you already know is down.
Next you will meet the senior-level failure mode: an agent that does damage not because the model misbehaved, but because it was handed more capability than its task ever needed.