Summarising the Conversation Threw Away the One ID the Next Tool Call Needed. Summarise Prose, Never Identifiers.

A support agent is forty turns into a refund conversation when the context gets compacted. The summary reads well: "Customer reported a damaged blender from their second order, agent confirmed eligibility, refund pending approval." Two turns later the agent calls issue_refund, and it needs an order ID that no longer exists anywhere in its context.

What happens next is the real problem. The agent rarely stops and says it lost something. It asks the customer for the number again, which feels broken. Or it calls search_orders(customer="…"), gets three results, and picks one. Or it writes an ID that looks right. Only the first of those is safe.

🔍 Why summaries drop exactly the wrong tokens

A summariser is asked to keep what matters and drop the rest. To a language model reading a transcript, meaning lives in the prose: who is upset, what broke, what was agreed. An opaque string like ord_8F3kQ2xM carries almost no narrative weight, so it gets compressed the same way a timestamp or a greeting does:

• Dropped: "the customer's second order" replaces the ID entirely.
• Generalised: "one of the recent orders" loses which one.
• Truncated: "order ord_8F3k…" keeps a prefix that matches nothing.
• Mutated: "ord_8F3kQ2xN" is one character off.

The mutated case is the dangerous one. It passes format validation. If your IDs are dense enough, it may even resolve to a different real record, and the refund lands on someone else's order with a perfectly plausible trace.

None of this means the summariser is bad. It did its job. Identifiers are not prose, and a lossy compressor for prose is the wrong container for them.

The fix: split the context into two lanes

Stop treating the transcript as one blob to shrink. Before compacting, separate it into:

1. Narrative — intent, decisions, open questions, tone. This goes to the summariser.
2. Identifiers — every ID, key, path, URL, account number, and handle the agent has actually touched. These go into a ledger that is copied forward verbatim and never passes through a model.

The summary then refers to ledger entries by role instead of by value.

def compact(messages, ledger):
    # 1. Harvest identifiers from structured tool results, not from prose.
    for msg in messages:
        if msg.role == "tool":
            for key, value in extract_ids(msg.structured_result):
                ledger.record(role=key, value=value, turn=msg.turn)

    # 2. Summarise narrative only, and tell the model the ledger exists.
    summary = llm.summarise(
        messages,
        instructions=(
            "Summarise decisions, intent and open items. "
            "Refer to entities by their ledger role, e.g. [damaged_order]. "
            "Never write an identifier value."
        ),
        ledger_roles=ledger.roles(),
    )

    # 3. Rebuild context: ledger first, byte-for-byte, then the summary.
    return [ledger.as_system_block(), summary, *messages[-KEEP_RECENT:]]

The ledger block the model sees afterwards is small and boring, which is the point:

ID LEDGER (verbatim, do not paraphrase)
damaged_order     ord_8F3kQ2xM      (seen turn 12, last used turn 37)
customer          cus_41ZpA9
refund_request    rfq_77Hn02        (status: pending_approval)

Now the summary says "refund for [damaged_order] pending approval" and the next tool call copies the value from a line that was never rewritten.

Details that decide whether this works

Harvest from tool results, not from chat text

The most reliable source of an identifier is the structured response of the tool that produced it. Pull fields by schema (order_id, ticket_id, file_path) rather than regex-scanning the conversation. Regex over prose will catch the customer typing a wrong number and promote it to ground truth.

If the user supplies an ID the agent has not yet verified with a tool, record it with a flag like unverified so the model knows not to act on it directly.

Give every entry a role, not just a value

A ledger that holds ord_8F3kQ2xM, ord_2Lp0Qe1R, ord_9Tt6Wb3C has swapped one problem for another: the agent now has the IDs but not which one was damaged. The role label is what the summary points at, so it has to exist.

Evict by last use, not first seen

Ledgers grow. When you trim, drop entries the agent has not referenced for the longest time, not the oldest ones. The order from turn 12 that is still being refunded on turn 40 is the one you most need.

Keep the recent turns raw

Compaction should never cover the last few turns. The tool call in flight almost always depends on something from the immediately preceding result, and there is no reason to put that through any transformation at all.

✅ How to test that compaction is safe

A summary that reads well to a reviewer tells you nothing about whether the agent can still act. Test the action instead:

• Take recorded traces from your evals.
• At every turn, force a compaction, then let the agent produce its next tool call.
• Diff the identifier arguments against the uncompacted run.

Any mismatch is a bug, even when the call still succeeds. A successful call with a wrong ID is the failure you are trying to catch.

Add one cheap runtime check as well: before executing a tool call, confirm every ID argument appears in the ledger or in the raw recent turns. If it appears in neither, the model produced it from memory. Reject the call and have the agent look it up again.

❌ Things that look like fixes and aren't

• "Preserve all IDs" in the summariser prompt. It helps, then fails on the long transcript where you needed it most. You are still asking a paraphraser to copy exact strings.
• Bigger context windows. They postpone compaction; they do not change what compaction does when it finally runs.
• Re-reading the full transcript from storage on demand. Workable, but it turns every follow-up into a retrieval problem with its own miss rate.

The takeaway

Compaction is two operations wearing one name. Compressing meaning is a job for a model, and it is allowed to be lossy. Carrying identifiers is a job for plain code, and it is not allowed to lose a single character. Once they sit in separate lanes, the summary can be as short as you like, and the next tool call still gets the order number it came for.