Vector Database or Postgres? Choosing Storage for Agent Memory

The default assumption is that agent memory needs a vector database. Sometimes it does. Frequently the queries an agent actually issues are filters and joins with a similarity component bolted on, and a relational database with a vector extension serves them better than a dedicated store.

The deciding factor isn't scale. It's the shape of the queries.

Look at the queries, not the data

Write down what your agent actually needs to retrieve. In practice it clusters into three shapes:

Pure similarity. "Find passages related to this question." No filters, no joins, ranking is everything. This is what dedicated vector stores are built for and where they excel.

Filtered similarity. "Find passages related to this question, from documents this user can access, written in the last year, excluding archived ones." Similarity plus predicates — and the predicates are often more selective than the similarity.

Structured lookup with no similarity at all. "What did we decide about the billing schema?" "What are this user's stated preferences?" Exact retrieval by key or category. A surprising share of agent memory reads are this, and embedding them is pure overhead.

Most agent systems are dominated by the second and third shapes, which is not what the "you need a vector DB" framing assumes.

Where each option is strong

Postgres with a vector extension wins on filtered similarity and structured lookup, because the filter and the join are native rather than metadata bolted onto an index. It also gives you transactions, which matter more than expected — memory writes usually need to happen atomically with something else, like marking a source processed.

The other underrated advantage: it's one system. One backup, one migration story, one place your agent's memories sit alongside the entities they refer to. A foreign key from a memory to the account it describes is worth a lot at debugging time.

A dedicated vector store wins on pure similarity at large scale, on ingest throughput for continuous embedding, and on the operational features built specifically for this — index tuning, hybrid search out of the box, sharding for very large collections. If your corpus is genuinely large and your queries are genuinely similarity-first, this is what it's for.

The scale question, honestly

Vector search in Postgres handles a lot more than people assume — well into the millions of vectors with appropriate indexing. Most agent memory stores are much smaller than that: a curated memory store is thousands to tens of thousands of records, not millions, because good memory is edited rather than accumulated.

⚠️ The trap is choosing infrastructure for a hypothetical corpus. If you're storing curated facts about users and entities, you're unlikely to reach the scale where a dedicated store's advantages bite, and you'll have paid the two-system tax the whole time.

Document search over a large corpus is a different workload from agent memory, and conflating them is how this decision gets made wrong. It's entirely reasonable to have both: Postgres for memory, a vector store for the document corpus.

What actually determines retrieval quality

Neither choice, mostly. In practice quality is determined by:

• What you store. Atomic, self-contained claims retrieve far better than passages, in any store.
• Hybrid retrieval. Combining keyword and vector search beats either alone on most agent workloads — identifiers, error codes, and product names are exact-match problems that embeddings handle poorly.
• Filtering before ranking. Narrowing to the eligible set, then ranking, beats ranking globally and filtering after.
• Recency handling. Similarity is indifferent to time; agent memory rarely should be.

✅ Every one of these is available in both options. Which is the real point: effort spent on what you store and how you query it returns more than effort spent choosing between the two.

A pragmatic default

Start with the database you already run, plus a vector extension. Model memories as rows with real columns — category, source, created_at, superseded_by, and an embedding.

Move to a dedicated store when you have a specific, measured reason: index build times hurting, similarity latency at your corpus size failing your budget, or a genuine document-search workload alongside memory. Those are recognizable when they arrive, and migrating a well-modelled memory store is much easier than un-migrating a two-system architecture you didn't need.

The takeaway

The question isn't vector-versus-relational, it's what your queries look like. Filtered similarity and structured lookup — most of agent memory — are relational workloads with a similarity component. Pure large-scale similarity is a vector store's job. Write down your ten most common retrievals before choosing, and put the saved effort into what you store rather than where you store it.