When the Context Window Fills, Cut in This Order — and Leave a Stub Behind

An HR assistant handles leave requests fine for six exchanges. On the seventh it quotes a policy that does not exist. Nothing crashed and no tool failed. The context window filled, something was dropped to make room, and the thing that got dropped was the part that mattered.

Every agent that runs longer than a few turns eventually evicts something. Most teams let the framework decide by dropping the oldest tokens. That treats the window as a queue, when it is actually four different kinds of content with wildly different costs.

🧱 Four kinds of context, not one blob

What sits in an agent's window at any moment is not homogeneous:

• Standing rules. Who the agent is, what it may and may not do, the escalation policy, the output format. Small, and catastrophic to lose.
• Task state. What the user asked, what has been decided, which steps are done. Grows steadily, mostly cheap prose wrapped around a few load-bearing facts.
• Retrieved facts. Things deliberately pulled in: a policy paragraph, an account record, a balance. Usually re-fetchable at the cost of one more call.
• Raw tool output. The 200-line API response, the HTML page, the log tail. The biggest thing in the window and the least dense.

They differ on two axes that nothing in a token counter can see: what it costs to lose, and what it costs to get back. Raw tool output is enormous and nearly free to re-derive. Standing rules are tiny and impossible to recover from inside the conversation. A length-based eviction policy is blind to both.

The priority order

Cut top to bottom, and stop as soon as the window fits:

1. Raw tool output older than the step that consumed it. Replace the dump with one line stating what it yielded.
2. Retrieved facts that are re-fetchable. Drop the text, keep the query or identifier that produced it.
3. Old task-state turns. Fold them into a running decision log: the conclusions, not the conversation that reached them.
4. Standing rules. Never. If eviction pressure is reaching these, the design is wrong, not the budget.

The order is not arbitrary. It ranks by loss cost divided by recovery cost, which is the only ratio that matters when you are choosing what to forget.

🔍 What naive truncation actually deletes

Walk the leave-request agent forward.

Turn 3, the agent establishes something important: this employee is on a fixed-term contract, so accrual follows a different schedule. That is nine words of task state.

Turn 5, it fetches the company holiday calendar. That returns a 4,000-token HTML table. It needed two dates from it.

Turn 9, the window is full. A tail-based truncation drops the middle of the conversation, because the system prompt is pinned at the top and the recent turns are obviously needed. The middle is turn 3.

So the nine words about the contract go, and the 4,000-token calendar table stays, because it is more recent. The agent recomputes accrual under the permanent-staff rule and states the wrong number with total confidence. No error surfaces anywhere. The user gets a number that is simply wrong.

Naive truncation evicted the highest-value, lowest-cost item in the window and kept the lowest-value, highest-cost one.

Replace, never delete

The fix that does most of the work is small: nothing leaves the window without leaving a stub behind.

def compact(window):
    for item in window.oldest_first():
        if window.fits():
            break
        if item.kind == "tool_output" and item.consumed:
            # keep the answer, drop the dump
            item.replace_with(item.result_line())
        elif item.kind == "retrieved" and item.refetchable:
            # keep the handle, drop the text
            item.replace_with(item.pointer())
        elif item.kind == "task_state":
            # keep the decision, drop the chat around it
            item.fold_into(window.decision_log)
        # standing_rules are never eligible

A stub for the calendar fetch is one line:

[tool holiday_calendar(region=UK, year=2026) -> used 2026-12-25, 2026-12-26; full table dropped at turn 7]

Twenty tokens instead of four thousand, and critically, the agent still knows the call happened and how to repeat it. Silent deletion produces an agent that does not know what it does not know. A stub produces one that can ask.

Three signals you cut the wrong thing

Watch for these in traces. Each maps to a specific eviction bug:

• The agent re-requests something it already received. → a retrieved fact was dropped without a pointer.
• It redoes a step that was already finished. → task state was cut without folding into a decision log.
• Its tone shifts, the output format drifts, or it offers something policy forbids. → standing rules fell out of the window. This is the serious one, and it looks like a model quality problem rather than a memory problem, which is why it goes undiagnosed for weeks.

⚠️ Do not optimise for tokens saved

Compaction that halves token count while doubling tool calls is a loss dressed as a win. The honest metric is a pair: tokens held, and re-fetch rate. If re-fetches climb after a compaction change, you cut into layer 2 or layer 3 when you should have been more aggressive at layer 1.

The window is not storage and it is not a queue. It is a working set with four tiers, and the only defensible eviction policy ranks them by what it costs to lose against what it costs to get back. Order it once, deliberately, and the seventh turn stops inventing policy.