Every payload from the last two lessons costs tokens, and the window is fixed. Sooner or later you have more candidate material than the window holds: a long conversation, a dozen retrieved chunks, a stack of tool definitions. Something has to give. Packing is the decision of what makes the cut — which parts to send and which to leave behind — so the model receives the highest-value text that fits. Get it wrong in either direction and you pay: send too much and you truncate or drown the instructions; send too little and the model answers without the facts it needed.
flowchart TD SYS["system instructions<br/>always first"] --> BOX["fixed token budget"] H["recent turns"] --> BOX C["retrieved chunks<br/>by relevance"] --> BOX O["older memory"] --> BOX style SYS fill:#1e7f3d,color:#fff style BOX fill:#8b5cf6,color:#fff
Priority before packing
Packing presumes an order, and the order is priority, not size. The standing instructions come first, because without them the model does not know the task or the format. The most recent turns come next, because the live question is what the user actually asked. Then come the retrieved chunks ranked by relevance, and only then older memory. You decide this order once, as policy; the packer then simply walks the list. Separating what matters (priority) from what fits (packing) keeps both decisions readable and testable.
Greedy, and keep going
The packer is deliberately simple: walk the priority-ordered list and take each part whole if it still fits in the remaining budget. The behaviour that matters most is what you do when a part does not fit. The right answer is to skip it and continue — a too-big retrieved chunk is passed over, and the smaller chunks after it still get their chance. This greedy walk fills the budget as fully as the available parts allow, never splitting a part, never reordering. It is not optimal in the mathematical sense, but it is predictable, fast, and exactly what most production retrievers do.
flowchart TD A["part fits?"] -->|"yes"| T["take it"] A -->|"no"| K["skip it"] T --> N["then try the next part"] K --> N style K fill:#b45309,color:#fff style T fill:#166534,color:#fff
function totalTokens(parts) {
let s = 0;
for (const p of parts) s += p.tokens;
return s;
}
const parts = [{ id: 'sys', tokens: 100 }, { id: 'turn', tokens: 60 }, { id: 'chunk', tokens: 40 }];
console.log('total:', totalTokens(parts));
function pack(parts, budget) {
let used = 0;
const chosen = [];
for (const p of parts) {
if (used + p.tokens <= budget) {
chosen.push(p.id);
used += p.tokens;
}
}
return chosen;
}
const parts = [{ id: 'sys', tokens: 100 }, { id: 'huge', tokens: 80 }, { id: 'small', tokens: 10 }];
console.log('packed:', pack(parts, 130));
The early-stop trap
The natural-looking mistake is to break out of the loop the first time a part does not fit. That feels safe — we hit something too big, stop — but it squanders the rest of the budget. Imagine a high-priority chunk that alone fills most of the window, followed by several small ones: a break keeps only the chunk and leaves the small ones behind, even though they would have fit in the space left over. The single keyword that separates a working packer from a wasteful one is continue, not break: miss one, try the next.
flowchart TD G["good packer<br/>skips and continues"] --> U["uses the<br/>whole budget"] B["buggy packer<br/>stops at first miss"] --> W["leaves budget<br/>unused"] style U fill:#166534,color:#fff style W fill:#b91c1c,color:#fff
function packReport(parts, budget) {
let used = 0;
const included = [], dropped = [];
for (const p of parts) {
if (used + p.tokens <= budget) { included.push(p.id); used += p.tokens; }
else { dropped.push(p.id); }
}
return { included, dropped, used };
}
const parts = [{ id: 'sys', tokens: 100 }, { id: 'huge', tokens: 80 }, { id: 'a', tokens: 15 }, { id: 'b', tokens: 10 }];
console.log(packReport(parts, 130));
Count tokens honestly
A packer is only as good as its token counts, and the honest unit is the token, not the character or the word. Two strings of equal length can be a different number of tokens, so estimate with the same tokenizer the model will use, or count characters and divide by a conservative ratio. Round each part's cost before packing and treat the budget as a hard ceiling with a little headroom for the model's reply. Off-by-one here is not a rounding quibble: a part you thought fit gets truncated mid-sentence, and a truncated instruction is worse than none.
What does this print? totalTokens adds up the tokens field of every part.
function totalTokens(parts) {
let s = 0;
for (const p of parts) s += p.tokens;
return s;
}
console.log(totalTokens([{tokens: 100}, {tokens: 50}, {tokens: 25}]));
Add the three tokens values together.
100 + 50 + 25.
Write totalTokens(parts) that returns the sum of the tokens field across every part in the array. parts looks like [{ id: 'a', tokens: 10 }].
function totalTokens(parts) {
// sum the tokens field
}
Start a running sum at 0.
Loop and add p.tokens each time.
function totalTokens(parts) {
let s = 0;
for (const p of parts) s += p.tokens;
return s;
}
Write pack(parts, budget) that greedily packs parts in the order given. Walk the list; if a part fits in the remaining budget (used + p.tokens <= budget), push its id into the result and add its tokens to used. If it does not fit, skip it and keep going — do not stop. Return the array of chosen ids.
function pack(parts, budget) {
// take each part that fits, skip the rest, keep going
}
Track used tokens and a result array.
When a part does not fit, do nothing and let the loop move on to the next part.
function pack(parts, budget) {
let used = 0;
const chosen = [];
for (const p of parts) {
if (used + p.tokens <= budget) {
chosen.push(p.id);
used += p.tokens;
}
}
return chosen;
}
This packer is meant to skip a part that doesn't fit and keep trying the rest, but it breaks out of the loop the first time a part is too big — so every smaller part after it is abandoned and the budget goes unused. Remove the early exit so the loop continues past parts that don't fit.
function pack(parts, budget) {
let used = 0;
const chosen = [];
for (const p of parts) {
if (used + p.tokens <= budget) {
chosen.push(p.id);
used += p.tokens;
} else {
break;
}
}
return chosen;
}
The else { break; } ends the whole walk at the first oversized part.
Delete that else branch entirely so the loop just skips non-fitting parts.
function pack(parts, budget) {
let used = 0;
const chosen = [];
for (const p of parts) {
if (used + p.tokens <= budget) {
chosen.push(p.id);
used += p.tokens;
}
}
return chosen;
}
Write packReport(parts, budget) that packs the same greedy way but returns an object with three fields: included (ids that fit), dropped (ids that did not fit), and used (total tokens of the included parts). Walk the list once, routing each id into included or dropped.
function packReport(parts, budget) {
// return { included: [...], dropped: [...], used: n }
}
Keep two id arrays and a used counter.
A part that fits goes into included and bumps used; otherwise it goes into dropped.
function packReport(parts, budget) {
let used = 0;
const included = [], dropped = [];
for (const p of parts) {
if (used + p.tokens <= budget) {
included.push(p.id);
used += p.tokens;
} else {
dropped.push(p.id);
}
}
return { included, dropped, used };
}
You are packing a priority-ordered list into a budget. The first part fits, the second is too big on its own, and the third is small. What should a correct greedy packer do?
A greedy packer continues past a part that does not fit, giving each later part its own chance. Stopping wastes the remaining budget, and splitting a part is never allowed.
Recap
- The window is finite: packing decides which priority-ordered parts make the cut.
- Priority, then packing: instructions first, recent turns next, retrieved chunks by relevance, older memory last.
- The packer is greedy: take each part whole if it fits, else skip and continue.
- The classic bug is
break— it abandons the budget the moment one part is too big. Usecontinue(or no else at all). - Count tokens honestly and leave headroom; a truncated instruction is worse than none.
With selection, structure, and packing in hand, the next lessons add few-shot examples and rolling memory.