When Memory Makes an Agent Worse

Memory is assumed to be an improvement — the agent remembers you, so it serves you better. Often true. But every retrieved memory is content injected into the context, and content in the context has effects whether or not it was relevant.

Three ways a memory system degrades the agent it was added to improve.

1. Stale memories outrank fresh input

A user says "actually, let's use USD for this one." The memory store contains "prefers EUR," retrieved and injected at the top of the context as an established fact about this user.

Now two statements conflict: a durable-looking preference presented as background truth, and a passing remark in the conversation. The framing favours the memory — it arrives labelled as something known about the user, while the correction is one sentence among many.

The fix is presentational as much as architectural. Memories should enter the context marked as defaults that current input overrides, and dated:

## Known preferences (defaults — the current conversation overrides these)
[since 2026-03] Reports in EUR

That single parenthetical changes behavior more than most prompt engineering. ⚠️ Without it, the agent argues with users about their own stated preferences, which is among the worst experiences a memory feature can produce.

2. Irrelevant memories dilute attention

A retriever configured to return the top five memories returns five memories regardless of whether any is relevant. On a request that needs none, five unrelated facts about the user enter the context, are re-sent every turn, and compete for attention with the actual task.

The cost is subtle: not a wrong answer traceable to a specific memory, but slightly worse performance across many requests. Hard to attribute, easy to miss.

Fixes: a relevance floor below which the store returns nothing, and an explicit empty result — "no relevant preferences found" — rather than a best-effort three. ✅ Track the ratio of retrieved memories that appear in the response; a low ratio means you're paying context for noise on every turn.

3. Confidently remembered wrong things

Memory writes are usually less scrutinized than anything else in the system. A misheard preference, an assumption recorded as a fact, an inference from one interaction promoted to a standing truth — and now it's retrieved and presented as known, indefinitely.

The compounding version: the agent acts on the wrong memory, the user doesn't correct it (they may not realize why the agent did that), and the memory persists with its apparent validity confirmed by not having been challenged.

Fixes: record the source and basis of every memory, so a wrong one can be traced back; require explicit statement or repeated observation before promoting something to a durable preference; and give users a way to see and correct what's remembered. A memory system with no user-visible surface is a system whose errors can never be reported.

🔍 Testing whether memory helps

Run your eval suite twice: memory enabled and disabled. Compare per-case, not in aggregate.

memory helped:     14 cases
memory neutral:    71 cases
memory hurt:        9 cases   ← read every one

Nine harmful cases against fourteen helpful ones is not the clear win an aggregate score suggests, especially since the harm often lands on requests where the user said something explicit and got overridden — the cases that annoy people most.

Read all nine. The pattern is usually one of the three above, and usually fixable.

What a healthy memory system looks like

• Small. Dozens to hundreds of curated facts, not thousands of turn fragments.
• Sourced. Every entry records where it came from and when.
• Dated on retrieval. Age is visible in context, so old claims arrive visibly old.
• Explicitly subordinate to the current conversation.
• Honest about empty. Returns nothing when nothing is relevant.
• Inspectable by the user, and correctable.

The takeaway

Memory is content injection, and content has effects regardless of relevance. Label memories as overridable defaults with dates, return nothing rather than the best of a bad set, record the basis of every entry, and let users see what's stored. Then run the A/B and read the cases where memory made things worse — there will be some, and they're concentrated exactly where a user told the agent something and was ignored.