Conversation Memory and Summarisation

Picture a support chat at turn forty. Every message the user has sent — their name, their order number, the thing they asked three minutes ago — has to ride along in the prompt for the model to answer coherently, because the model has no memory between calls. Each turn re-sends the whole history. So the conversation grows, and grows, until it no longer fits the context window.

The lazy fix is truncation: keep only the last few messages and throw the rest away. It is cheap, and it is catastrophic, because the oldest messages are exactly where the durable facts live — my name is Ada, I ordered a blue one. Lose those and the model starts asking the user's name again. This lesson builds the real answer: a rolling-summary memory that keeps the recent turns verbatim and compresses the older ones into a summary, so a long conversation stays inside budget without forgetting who it is talking to.

flowchart TD
  H["full conversation history"] --> N["naive truncation"]
  N --> L["keeps tail only<br/>early facts lost"]
  H --> R["rolling summary memory"]
  R --> K["recent turns verbatim<br/>older turns summarised"]
  style L fill:#b91c1c,color:#fff
  style K fill:#166534,color:#fff

Naive truncation keeps only the tail and loses early facts; a rolling summary preserves them in compressed form.

The rolling-summary mechanism

Split the conversation into two regions. The recent window holds the last N turns word for word — these matter most, because the model needs exact phrasing to continue the thread. Everything older flows into a running summary, a compressed paraphrase that gets one new sentence folded in each time a turn ages out of the window.

The mechanics are pure bookkeeping. Each incoming message is appended to the recent window. The moment the window grows past its limit, the oldest message is shifted out and folded into the summary, and the window shrinks back to size. The prompt the model sees is then two pieces stitched together: the running summary on top, the recent window below. Nothing is re-sent in full once it has been compressed; the cost is bounded by the window size plus the summary's length, not by how old the conversation is.

Approximate tokens by word count, then watch naive truncation drop the user's name. Run it.

function countTokens(text) {
  const t = text.trim();
  if (t === "") return 0;
  return t.split(/\s+/).length;
}

const convo = ["My name is Ada", "I like tea", "What is 2+2?", "Thanks"];
const kept = convo.slice(-2);                  // naive: keep only the tail
console.log("tokens kept:", countTokens(kept.join(" ")));
console.log(kept.join(" | "));                  // the name is already gone

The summary is lossy by design

Compression trades exact words for fewer tokens, which means the summary is always an approximation. A turn that read I ordered SKU 4419, the blue one, on Tuesday might become user ordered a blue item. The gist survives; the exact identifiers may not. That is the central trade of this whole technique: you buy a bounded, predictable prompt size by accepting that old detail is paraphrased rather than quoted.

Because no model is called in these exercises, the summarise step here is a deterministic stand-in — the aged message is moved into a summary bucket verbatim. In a real system that step is a small model call, but the bookkeeping around it (when to compress, what to keep, how to reassemble the prompt) is exactly the deterministic logic you build and test below.

flowchart LR
  W["recent window<br/>size N"] -->|"new message arrives"| O["over capacity"]
  O -->|"shift oldest out"| S["fold into running summary"]
  S --> P["summary and window form the prompt"]
  style O fill:#b45309,color:#fff
  style S fill:#8b5cf6,color:#fff

When the window overflows, the oldest message shifts out and folds into the running summary.

A rolling memory: recent turns stay verbatim, overflow moves into a summary. Run it.

function makeMemory(windowSize) {
  const recent = [];
  const dropped = [];
  return {
    push(msg) {
      recent.push(msg);
      while (recent.length > windowSize) dropped.push(recent.shift());
    },
    context() {
      const win = recent.join(" | ");
      if (dropped.length === 0) return win;
      return "Summary: " + dropped.join(" ... ") + " || " + win;
    }
  };
}

const mem = makeMemory(2);
["My name is Ada", "I like tea", "What is 2+2?"].forEach(m => mem.push(m));
console.log(mem.context());   // the name survives in the summary

The budget becomes bounded

The payoff of all this bookkeeping is that the prompt size stops growing with the conversation's age. The model's prompt is now two regions of bounded size: the recent window is at most N turns, and the summary is at most one short paragraph — if it ever grows, you can re-summarise the summary itself. Add them and you have a ceiling you can compute ahead of time, which means a forty-turn chat and a four-hundred-turn chat cost roughly the same input tokens.

That predictability is what lets you reserve a known slice of the context window for retrieved documents and instructions, without guessing whether the history will swallow it. Memory stops being a budget threat and becomes a fixed line item.

Check whether a set of parts fits a token budget before assembling the prompt. Run it.

function countTokens(text) {
  const t = text.trim();
  if (t === "") return 0;
  return t.split(/\s+/).length;
}

function fitsBudget(parts, budget) {
  const total = parts.reduce((s, p) => s + countTokens(p), 0);
  return total <= budget;
}

console.log(fitsBudget(["hello world", "foo bar baz"], 6));   // 5 tokens <= 6
console.log(fitsBudget(["hello world", "foo bar baz"], 4));   // 5 tokens > 4

flowchart LR
  P["prose: feelings, what they tried"] --> Sum["running summary<br/>paraphrased"]
  E["exact values: IDs, numbers, names"] --> Facts["facts store<br/>verbatim"]
  style Sum fill:#8b5cf6,color:#fff
  style Facts fill:#166534,color:#fff

Prose context can be paraphrased into the summary; exact values must route to a separate facts store.

What must never go into the summary

Some facts lose everything when paraphrased. An account ID, a confirmation number, a price, a date, a person's exact name — paraphrase these and they stop being usable. The professional pattern gives them their own home: a small structured facts store keyed by type, written once and never compressed. The rolling summary handles the conversational flow; the facts store holds the exact values the model must quote back.

The rule of thumb is simple. If the value is prose — how the user feels, what they tried — the summary may paraphrase it. If the value is a token the system must reproduce exactly — an ID, a number, a code — pull it into structured storage before it can be summarised away.

Exercise

What does this print? Word count is a rough proxy for token count.

function tokens(text) {
  return text.trim().split(/\s+/).length;
}
console.log(tokens("the quick brown fox"));

Your prediction — what does this print?

Exercise

Write countTokens(text) that returns a rough token count as the number of whitespace-separated words. An all-whitespace or empty string must return 0 (not 1).

function countTokens(text) {
  // return the word count; 0 for empty/whitespace-only input
}

function countTokens(text) {
  const t = text.trim();
  if (t === "") return 0;
  return t.split(/\s+/).length;
}

Exercise

This trimWindow keeps a conversation's recent window at maxKeep messages, but it drops the newest message (pop) instead of the oldest. In a conversation you must keep the recent context, so the oldest should go. Fix the one method call.

function trimWindow(messages, maxKeep) {
  while (messages.length > maxKeep) {
    messages.pop();
  }
  return messages;
}

function trimWindow(messages, maxKeep) {
  while (messages.length > maxKeep) {
    messages.shift();
  }
  return messages;
}

Exercise

A user states their account ID early in a long chat managed by rolling-summary memory. What is the safest way to keep that ID available?

Exercise

Write makeMemory(windowSize) returning an object with push(msg) and context(). push keeps at most windowSize recent messages verbatim; any overflow is moved (oldest first) into a summary bucket. context() returns just the recent window joined by " | " when nothing has been dropped, otherwise "Summary: " + dropped messages joined by " ... " + " || " + the recent window.

function makeMemory(windowSize) {
  // keep a recent window and a summary bucket of dropped messages
}

function makeMemory(windowSize) {
  const recent = [];
  const dropped = [];
  return {
    push(msg) {
      recent.push(msg);
      while (recent.length > windowSize) dropped.push(recent.shift());
    },
    context() {
      const win = recent.join(" | ");
      if (dropped.length === 0) return win;
      return "Summary: " + dropped.join(" ... ") + " || " + win;
    }
  };
}

Recap

Every turn re-sends the whole conversation, so a long chat overflows the context window. Naive truncation keeps the tail and loses the oldest facts.
A rolling-summary memory splits history into a recent window (verbatim) and a running summary (compressed older turns); when the window overflows, the oldest message folds into the summary.
The summary is lossy by design — it preserves gist, not exact words. The bookkeeping of when and what to compress is deterministic logic you can test.
Trim the oldest, never the newest: use shift, not pop, so the recent context stays.
The budget becomes bounded: window plus summary has a computable ceiling, so a long chat costs roughly the same tokens as a short one.
Pull exact values — IDs, numbers, names — into a separate facts store before they can be paraphrased, and summarise lazily to keep drift rare.

Next you will pack instructions, retrieved chunks, memory, and examples into one window and decide what stays.

Conversation Memory and Summarisation

The rolling-summary mechanism

The summary is lossy by design

The budget becomes bounded

What must never go into the summary

Recap

Checkpoint quiz

Go deeper — technical resources