An AI request is never one thing. It is a pipeline of steps: the input is parsed, context is retrieved, a tool is called, and the result is formatted into an answer. When that pipeline fails in production, the only signal a plain console.log gives you is a single line printed just before the crash. Which step broke? Why? You re-run the request hoping to reproduce it, and sometimes you cannot.
Observability is the practice of recording enough about a request to answer those questions after the fact, without re-running anything. The unit it records in is the span: a structured record of one step, carrying its name, its outcome, and the data it touched. A trace is the collection of spans a single request produced. Read the trace and the failing step names itself. This lesson builds spans by hand so the mechanism stops being magic.
flowchart LR REQ["request"] --> SP1["span: parse"] SP1 --> SP2["span: retrieve"] SP2 --> SP3["span: tool_call"] SP3 --> SP4["span: format"] SP4 --> RESP["response"] style REQ fill:#3776ab,color:#fff style RESP fill:#166534,color:#fff
What a span holds
A span is a plain object with a fixed shape. It carries a name for the step, a status that moves through a lifecycle, an attributes object for structured data, and an error field. The status begins at running the instant the span is created and ends at either ok or error when the step finishes. A span still marked running at the end of a request is itself a red flag: a step was opened and never closed, almost always because an exception escaped around the code that should have closed it. A trace is just a list of these spans plus the id of the request that produced them.
function startSpan(trace, name) {
const span = { name, status: 'running', attributes: {}, error: null };
trace.spans.push(span);
return span;
}
function endSpan(span, status) { span.status = status; }
const trace = { spans: [] };
const parse = startSpan(trace, 'parse');
parse.attributes.tokens = 4;
endSpan(parse, 'ok');
console.log(trace.spans.length);
console.log(trace.spans[0].name + ': ' + trace.spans[0].status);
Attributes are the evidence
The attributes object is where a span stores the detail you will need later. A retrieve span might record { docs: 3, topScore: 0.82 }; a tool_call span stores the result it returned. Because attributes are free-form key-value pairs, a span carries exactly the data that lets you diagnose a failure without re-running the request. The error field stays null until a thrown exception populates it, so a span with a non-null error is an unambiguous, machine-checkable flag that this step is where things went wrong.
Wrapping a step in a span
You will not sprinkle manual start and end calls around every line. That is exactly how a step gets missed. Instead wrap a whole step in a helper that opens the span, runs the step, and closes it with the correct status on every path. On success it records the result in attributes and marks the span ok. On failure it writes the error and marks it error. The rule that keeps this honest is record, then re-throw: the trace must show the failure, but the caller must still receive the exception. A span is a witness to what happened, not a try/catch that hides it. Close the span on the failing path, and then let the error travel back out.
flowchart TD S["span: status = running"] --> OK["end -> status = ok"] S --> ERR["throw -> status = error<br/>error recorded"] style OK fill:#166534,color:#fff style ERR fill:#b91c1c,color:#fff
function startSpan(trace, name) {
const span = { name, status: 'running', attributes: {}, error: null };
trace.spans.push(span);
return span;
}
function endSpan(span, status) { span.status = status; }
function traced(trace, name, fn) {
const span = startSpan(trace, name);
try {
const result = fn();
span.attributes.result = result;
endSpan(span, 'ok');
return result;
} catch (e) {
span.error = String(e);
endSpan(span, 'error');
throw e;
}
}
function findError(trace) {
return trace.spans.find(s => s.status === 'error') || null;
}
const trace = { spans: [] };
traced(trace, 'parse', () => 'parsed ok');
try {
traced(trace, 'retrieve', () => { throw 'db timeout'; });
} catch (e) {
// the request failed, but the trace recorded which step and why
}
const culprit = findError(trace);
console.log(culprit.name);
console.log(culprit.error);
Diagnosing from a trace
Once every step records a span, diagnosing a failure stops being guesswork. Given a trace, you scan its spans for the first one whose status is error. That is the step that broke, and its error field holds the reason. You did not have to reproduce the request, attach a debugger, or wait for the failure to happen again; the trace captured the moment. The same scan generalises to scale: count spans by status across a thousand traces, and a step that errors one time in twenty shows up as a steady trickle of error spans, which is exactly how a flaky dependency announces itself before a user reports it.
flowchart TD A["parse: ok"] --> B["retrieve: ok"] B --> C["tool_call: error"] C --> D["format: skipped"] FE["findError(trace)"] -.->|"returns"| C style C fill:#b91c1c,color:#fff style FE fill:#b45309,color:#fff
What does this print? A span is created with status running, then its status is set to ok.
const span = { name: 'parse', status: 'running', attributes: {}, error: null };
span.status = 'ok';
console.log(span.name + ': ' + span.status);
The span's name is 'parse' and status is changed to 'ok'.
The two are joined with ': ', giving 'parse: ok'.
Write traced(trace, name, fn) that starts a span (via the provided startSpan), runs fn(), and on success stores the result in span.attributes.result and ends the span with status ok. On a thrown error it stores String(e) in span.error, ends the span with status error, and then re-throws the error. startSpan and endSpan are already defined for you.
function startSpan(trace, name) {
const span = { name, status: 'running', attributes: {}, error: null };
trace.spans.push(span);
return span;
}
function endSpan(span, status) { span.status = status; }
function traced(trace, name, fn) {
// start a span, run fn(); on success store result + status 'ok';
// on error store the error + status 'error', then re-throw
}
Open the span with startSpan, then wrap fn() in try / catch.
On success: store result, endSpan 'ok', and return the result.
On error: store String(e), endSpan 'error', then throw e so the caller still sees it.
function startSpan(trace, name) {
const span = { name, status: 'running', attributes: {}, error: null };
trace.spans.push(span);
return span;
}
function endSpan(span, status) { span.status = status; }
function traced(trace, name, fn) {
const span = startSpan(trace, name);
try {
const result = fn();
span.attributes.result = result;
endSpan(span, 'ok');
return result;
} catch (e) {
span.error = String(e);
endSpan(span, 'error');
throw e;
}
}
This traced catches the error, records it on the span, and then swallows it by returning undefined. The trace looks correct, but the caller never learns the step failed, so the request silently returns nothing. Fix it so the error is re-thrown after it is recorded.
function startSpan(trace, name) {
const span = { name, status: 'running', attributes: {}, error: null };
trace.spans.push(span);
return span;
}
function endSpan(span, status) { span.status = status; }
function traced(trace, name, fn) {
const span = startSpan(trace, name);
try {
const result = fn();
span.attributes.result = result;
endSpan(span, 'ok');
return result;
} catch (e) {
span.error = String(e);
endSpan(span, 'error');
return undefined; // BUG: swallows the error
}
}
The error is already recorded on the span; only the swallow is wrong.
Replace
return undefined;withthrow e;so the caller still sees the failure.
function startSpan(trace, name) {
const span = { name, status: 'running', attributes: {}, error: null };
trace.spans.push(span);
return span;
}
function endSpan(span, status) { span.status = status; }
function traced(trace, name, fn) {
const span = startSpan(trace, name);
try {
const result = fn();
span.attributes.result = result;
endSpan(span, 'ok');
return result;
} catch (e) {
span.error = String(e);
endSpan(span, 'error');
throw e;
}
}
Write findError(trace) that returns the first span in trace.spans whose status is 'error', or null if none failed. This is how you point at the failing step after the fact.
function findError(trace) {
// return the first span whose status is 'error', or null
}
Loop over trace.spans and return the first span whose status is 'error'.
If the loop finishes without finding one, return null.
function findError(trace) {
for (const s of trace.spans) {
if (s.status === 'error') return s;
}
return null;
}
Write spanValue(trace, name) that finds the span with the given name and returns its recorded result (span.attributes.result), or undefined if no span has that name. This is how you recover the data a step captured without re-running it.
function spanValue(trace, name) {
// return the recorded result of the span with this name, or undefined
}
Loop over trace.spans looking for the span whose name matches.
Return its attributes.result when found; return undefined otherwise.
function spanValue(trace, name) {
for (const s of trace.spans) {
if (s.name === name) return s.attributes.result;
}
return undefined;
}
Recap
- A request is a pipeline; when it fails, a flat log line cannot tell you which step broke or why.
- A span records one step:
name,status(runningthenokorerror),attributes, anderror. A trace is the list of spans from one request. - Wrap each step in a helper that opens the span, runs it, and closes it with the right status on both paths.
- Record, then re-throw: write the error on the span, then throw it back out so the caller still sees the failure. Never swallow.
- Read a trace back with helpers:
findErrorpoints at the failing step, and itsattributescarry the data you need, with no re-run required.
You now have the last diagnostic tool. The capstone assembles these pieces into one platform you can grade end to end.