Chunking for Agents Is Not Chunking for Search

Retrieval advice mostly comes from search: split documents into passages, embed, rank, return the top few. It's good advice for search, where a person reads the results and supplies the missing context themselves. An agent doesn't. It acts on what comes back, and passages optimized for human scanning produce a specific set of agent failures.

What's different about the consumer

A person reading search results tolerates a fragment. They see it's a fragment, click through if it matters, and mentally note that a sentence beginning "This means the limit is 60" refers to something above it.

An agent receives that same fragment as a fact in its context, with no signal that it's incomplete and no reliable instinct to fetch the surrounding text. Then it acts.

Three consequences follow.

1. Chunks must be self-contained claims

A passage that depends on its neighbours for meaning is a hazard. Pronouns without referents, "the above table," "this setting," a threshold with no subject.

The fix isn't larger chunks — it's resolved chunks. When indexing, expand references so each passage stands alone:

before: "This means the limit is 60 requests per minute."
after:  "[Standard tier, rate limits] The Standard tier limit is
         60 requests per minute."

Prepending the document title and section path is the cheapest version of this and fixes a surprising amount. Rewriting the passage to resolve references costs an indexing-time model call and fixes the rest.

2. Retrieved material needs provenance attached

For search, the link is next to the result. For an agent, the source has to travel with the text or it's lost by the time synthesis happens.

Every retrieved chunk should arrive carrying its document ID, section, and date. This isn't only for citation — it lets the agent weigh sources against each other, and it lets you debug a wrong answer back to the passage that caused it.

3. Structure matters more than similarity for structured content

Tables, code, configuration, and procedures break badly under fixed-size splitting. Half a table is worse than no table: the column headers are in one chunk and the rows in another, and the rows alone are meaningless numbers the agent will happily use.

For structured content, split on structural boundaries — whole table, whole function, whole procedure — even where that means uneven chunk sizes. ⚠️ Uneven chunks are a minor inconvenience for the index and a major improvement in what the agent receives.

The agent-specific retrieval moves

Beyond chunking, three things help specifically because the consumer is an agent:

Return a small number of high-quality chunks, not the top twenty. Every chunk costs context that's re-sent on every subsequent turn. In search, a marginal result at rank eight is free to ignore. In an agent, it's paid for repeatedly and it dilutes attention.

Make "nothing relevant found" a real answer. A retriever always returning its top three returns three irrelevant passages when nothing matches, and an agent tends to use what it's given. A relevance floor, below which the tool returns an explicit empty result with a reason, prevents a whole class of confidently wrong answers.

Offer follow-up retrieval as a tool. get_surrounding_context(chunk_id) and get_full_document(doc_id) let an agent do what a person does with a search result — go and read more when the fragment isn't enough. Without these, its only option is to act on the fragment.

💡 Hybrid retrieval matters more here

Agent queries contain exact tokens far more often than search queries do: error codes, ticket IDs, function names, product SKUs, config keys. Embeddings handle these poorly — semantically, ERR_4471 and ERR_4472 are nearly identical.

Combining keyword and vector retrieval isn't a refinement in agent systems; it's usually the difference between finding the right record and finding a plausible neighbour. If your agent deals in identifiers at all, exact matching needs to be in the path.

🔍 Evaluating it

Don't evaluate retrieval on ranking metrics alone. The question that matters is whether the agent got the answer right, so:

• Write cases where the answer requires a specific passage, and assert the final output contains it.
• Write cases where nothing relevant exists, and assert the agent says so rather than using the closest match.
• Write cases requiring two passages from different documents, and assert both were used — multi-hop is where chunking quality shows up most sharply.

The takeaway

Chunking for search optimizes for a human who fills in the gaps. Chunking for an agent has to optimize for a consumer that doesn't. Make chunks self-contained by resolving references and prepending context, split structured content on structural boundaries, attach provenance, return few chunks rather than many, allow an honest empty result, and give the agent tools to read more. Then evaluate on whether the agent was right, not on whether the ranking looked good.