You write a perfect prompt, hit send, and the API throws a 400: context length exceeded. Nothing in your code changed. What happened is that the running conversation finally grew past the model's context window — the fixed number of tokens it can attend over in a single call. The window is a hard ceiling, not a suggestion, and every token you have ever sent in this conversation is still inside it: the system instructions, the retrieved documents, the full back-and-forth history, plus the room the model needs to write its reply. When the sum crosses the line, something has to go, and the only question is what — and who decides. This lesson is the accounting that answers that. You will count a token budget, reserve space for output, and implement the truncation rule that decides what a model can and cannot see.
What fills the window
A context window is not one bucket; it is a budget split across several claimants that all compete for the same tokens. There is the system prompt — the standing instructions that set the model's behaviour — present on every call. There is the conversation history, the prior user and assistant turns, which grows with every reply. There may be retrieved context, the documents a RAG pipeline fetched for this particular question. There is the current user message itself. And there is the output reservation, the tokens set aside so the model actually has room to answer. Add them and you get the total the model must hold at once. The window is the upper bound on that total, and the painful fact is that history and retrieved text grow without bound while the window does not.
flowchart LR W["context window"] --> A["system prompt"] W --> H["conversation history"] W --> R["retrieved docs"] W --> U["user message"] W --> O["output reservation"] style W fill:#3776ab,color:#fff style O fill:#b45309,color:#fff
What falls off the edge
When the total exceeds the window you have to cut, and the sensible cut is almost always the oldest history. The reasoning is simple: the most recent turns are what the model needs to answer the current question, while the earliest turns are the stalest context. A keep-recent strategy walks the message list from the newest backward, keeping each message while it still fits the remaining budget and stopping the moment the next one would overflow. Everything older than that cutoff simply falls off the edge — the model no longer sees it, as though it were never said. This is why a long chat appears to forget its earliest messages: it did not forget, it was truncated. On this call the model never saw those tokens, because you chose not to send them.
function totalTokens(messages) {
return messages.reduce((sum, m) => sum + m.tokens, 0);
}
function fitsWithin(messages, window) {
return totalTokens(messages) <= window;
}
const chat = [
{ role: "system", text: "be brief", tokens: 5 },
{ role: "user", text: "hi", tokens: 2 },
{ role: "assistant", text: "hello", tokens: 3 }
];
console.log(totalTokens(chat)); // 10
console.log(fitsWithin(chat, 8)); // false
console.log(fitsWithin(chat, 10)); // true
The reservation teams forget
The one budget item teams forget most often is the output reservation. The window covers input and output together: the tokens the model emits as its answer come out of the same fixed pool. If you pack the input right up to the limit, the model has nowhere to write and either truncates mid-sentence or errors out. The fix is to subtract a reservation before you ever budget the input: decide how many tokens the reply needs — 512, 1024, whatever your feature requires — and treat the usable input budget as the window minus that reservation. Then your truncation logic works against the real, smaller ceiling. Skipping this step is the difference between a feature that works in the playground, where you never approach the limit, and one that breaks the first time a real conversation grows long.
flowchart LR M1["msg 1 oldest"] -.-> X["dropped"] M2["msg 2"] -.-> X M3["msg 3"] --> K["kept: fits budget"] M4["msg 4 newest"] --> K style X fill:#b91c1c,color:#fff style K fill:#166534,color:#fff
function truncate(messages, window, reserveOutput) {
const budget = window - reserveOutput;
let used = 0, count = 0;
for (let i = messages.length - 1; i >= 0; i--) {
if (used + messages[i].tokens > budget) break;
used += messages[i].tokens;
count++;
}
return messages.slice(messages.length - count);
}
const chat = [
{ role: "user", text: "old", tokens: 10 },
{ role: "user", text: "mid", tokens: 10 },
{ role: "user", text: "new", tokens: 10 }
];
console.log(truncate(chat, 25, 0).length); // budget 25 -> keeps 2 newest
console.log(truncate(chat, 25, 10).length); // reserve 10 -> budget 15 -> keeps 1
Assembling a real prompt
Real prompts are not just a flat history; they are built from parts that have different priorities. The system prompt is sacred — drop it and the model loses its instructions — and the user's latest question is the whole point of the call, so both are pinned: they go in no matter what. Everything between them, the conversation history, is what you trim to fit. The assembly rule is therefore: always include the system prompt and the current query, then fill the remaining budget with as many recent history turns as fit, newest first. That yields a prompt that always answers the user, always obeys the system, and gracefully sheds the oldest context the moment the conversation outgrows the window.
flowchart TD P["input fills the whole window"] --> E["no room for output"] E --> B["reply truncated or error"] Q["reserve output tokens first"] --> S["smaller safe input budget"] S --> G["reply completes"] style B fill:#b91c1c,color:#fff style G fill:#166534,color:#fff
function composePrompt(system, history, query, window, reserve) {
const budget = window - reserve;
let remaining = budget - system.tokens - query.tokens;
const kept = [];
for (let i = history.length - 1; i >= 0; i--) {
if (history[i].tokens <= remaining) {
remaining -= history[i].tokens;
kept.unshift(history[i]);
} else {
break;
}
}
return [system, ...kept, query];
}
const sys = { role: "system", text: "be brief", tokens: 10 };
const q = { role: "user", text: "summary?", tokens: 10 };
const hist = [
{ role: "user", text: "a", tokens: 5 },
{ role: "user", text: "b", tokens: 5 },
{ role: "user", text: "c", tokens: 5 }
];
console.log(composePrompt(sys, hist, q, 30, 0).length); // pins 20 + 2 hist (10) = 4
What does this print? totalTokens adds up the tokens field of every message.
function totalTokens(messages) {
return messages.reduce((sum, m) => sum + m.tokens, 0);
}
const chat = [
{ role: "system", text: "s", tokens: 5 },
{ role: "user", text: "u", tokens: 7 },
{ role: "assistant", text: "a", tokens: 3 }
];
console.log(totalTokens(chat));
Add the tokens field of each message: 5, then 7, then 3.
5 + 7 + 3 = 15.
Write totalTokens(messages) that returns the sum of every message's tokens field. Each message is an object with a numeric tokens property.
function totalTokens(messages) {
// sum m.tokens over all messages
}
Use reduce with a starting accumulator of 0.
Add m.tokens to the running sum for each message.
function totalTokens(messages) {
return messages.reduce((sum, m) => sum + m.tokens, 0);
}
Write truncate(messages, window, reserveOutput) that keeps the most recent messages that fit. Messages are ordered oldest to newest. First subtract reserveOutput from window to get the input budget, then take messages from the end (newest first) while they fit, stopping at the first one that would overflow. Return the kept messages in their original order.
function truncate(messages, window, reserveOutput) {
// budget = window - reserveOutput; keep a recent suffix that fits
}
Compute budget = window - reserveOutput, then walk from the last index backward.
Accumulate tokens; break when the next message would exceed the budget. Slice off the last
countmessages.
function truncate(messages, window, reserveOutput) {
const budget = window - reserveOutput;
let used = 0, count = 0;
for (let i = messages.length - 1; i >= 0; i--) {
if (used + messages[i].tokens > budget) break;
used += messages[i].tokens;
count++;
}
return messages.slice(messages.length - count);
}
truncate must subtract reserveOutput from window before deciding what fits, because input and output share one window. This version ignores the reservation and budgets against the raw window, so it keeps too much and leaves the reply nowhere to land. Fix the budget line.
function truncate(messages, window, reserveOutput) {
const budget = window; // BUG: ignores reserveOutput
let used = 0, count = 0;
for (let i = messages.length - 1; i >= 0; i--) {
if (used + messages[i].tokens > budget) break;
used += messages[i].tokens;
count++;
}
return messages.slice(messages.length - count);
}
The usable input budget is window minus reserveOutput.
Change
const budget = windowto subtract reserveOutput.
function truncate(messages, window, reserveOutput) {
const budget = window - reserveOutput;
let used = 0, count = 0;
for (let i = messages.length - 1; i >= 0; i--) {
if (used + messages[i].tokens > budget) break;
used += messages[i].tokens;
count++;
}
return messages.slice(messages.length - count);
}
Write composePrompt(system, history, query, window, reserve) that assembles the final message list. Always place system first and query last. Then fit as many recent history messages as the leftover budget allows: budget = window - reserve, then subtract system.tokens and query.tokens. Take history from the end while each fits, stopping at the first that does not. Return [system, ...keptHistory, query].
function composePrompt(system, history, query, window, reserve) {
// pins: system + query; fill the rest with recent history that fits
}
budget = window - reserve, then remaining = budget - system.tokens - query.tokens.
Walk history from the end; keep each message while its tokens fit
remaining, subtract them, and break at the first that does not.
function composePrompt(system, history, query, window, reserve) {
const budget = window - reserve;
let remaining = budget - system.tokens - query.tokens;
const kept = [];
for (let i = history.length - 1; i >= 0; i--) {
if (history[i].tokens <= remaining) {
remaining -= history[i].tokens;
kept.unshift(history[i]);
} else {
break;
}
}
return [system, ...kept, query];
}
Recap
- The context window is a hard ceiling on the tokens a model can attend over in one call; it is shared by input and output together.
- The window is a budget split across system prompt, conversation history, retrieved docs, the user message, and the output reservation.
- When the total overflows, a keep-recent strategy drops the oldest messages first — they fall off the edge and are simply not sent on that call.
- Always compute the input budget as window minus reserveOutput; forgetting the reservation works in the playground and breaks the first long conversation in production.
truncatekeeps a recent suffix that fits;composePromptpins the system prompt and the current query and fills the leftover budget with recent history.- A long chat that forgets early messages was not forgetful — it was truncated by your own budget logic.
Next you will turn this accounting into a cost model: separating input from output token pricing and exposing where real spend actually hides.