One Query Is Rarely Enough: Iterative Retrieval for Agents

A support agent was asked why the checkout latency incident in March was closed without a fix. It ran one search on the question text and got back three documents about checkout, two about latency generally, and nothing about that incident. It answered from what it had, confidently and wrongly.

The retrieval wasn't broken. It answered the question as typed, which is a different thing from the question asked — and a single search can only ever do the first.

Why one query fails

Vocabulary mismatch. The user says "closed without a fix." The incident record says "resolved — no action required, transient upstream." No shared terms, and embeddings only partly bridge the gap when the phrasing diverges this far.

Multi-hop dependencies. Finding the incident requires knowing its ID. The ID lives in a document you find by searching for something else. One query cannot traverse that — it needs the result of the first search to construct the second.

Conjunctive questions. "Which customers on the legacy plan opened tickets about the March outage?" is two retrievals and a join, dressed as one sentence. A single embedding of the whole question matches documents that are vaguely about both and precisely about neither.

Filter-shaped constraints. "In March" is a date filter, not a semantic feature. Embedded into the query it contributes almost nothing, and results from any month rank equally.

Retrieval as a turn, not a preprocessing step

The standard pipeline retrieves once and then generates. That treats search as something done to the agent before it thinks.

The change worth making: retrieval is a tool the agent calls, sees the result of, and calls again. Then it can react — notice that nothing relevant came back, extract an identifier and search for it, split a compound question, loosen a filter.

This only works if the search tool is honest about what it returned. A retriever that always hands back its top three gives the agent no signal that the search failed, so the agent proceeds as though the record is complete. The tool needs to be able to say nothing matched:

search(query, filters, k) -> {
  results: [{doc_id, chunk_id, score, text, source, date}],
  total_matched: 41,
  returned: 5,
  max_score: 0.31,
  note: "No result above the relevance threshold (0.45).
         Try different terms, or widen the date filter."
}

→ max_score and an explicit below-threshold note are what turn a failed search into a next action rather than a wrong answer.

Four moves worth supporting

Reformulate. After a weak result, search again with different vocabulary — the domain's terms rather than the user's. Support this by returning the note above, and by prompting explicitly that a low-scoring result means the query was wrong, not that the answer doesn't exist.

Decompose. Split a conjunctive question into separate searches and combine. The agent does this well when told it may, and rarely thinks of it otherwise.

Follow the entity. The first search surfaces INC-4471; the second searches for that ID directly. This is where hybrid retrieval matters — an identifier is an exact-match problem, and embeddings rank INC-4471 and INC-4472 as near-identical. Without keyword search in the path, entity-following silently returns the wrong record.

Adjust the filters. Filters as real parameters, so a search returning nothing can be retried wider and one returning hundreds can be narrowed. Expose the fields — date, source, status — rather than expecting the agent to encode them in the query text.

⚠️ Iterative retrieval loops

The failure this pattern introduces: an agent that searches, gets nothing, rephrases slightly, gets nothing, and repeats. Each iteration is a model call carrying a growing context.

Three controls:

• A search budget per run, visible to the agent, so it prioritizes as it depletes.
• Query deduplication. Keep the issued queries; refuse a near-identical repeat with a message saying so. Agents rephrase cosmetically far more often than substantively.
• A stop rule that produces a useful answer. On exhausting the budget, the agent should report what it searched for and what it couldn't find — not guess. "I searched for the March checkout incident by date, by service, and by ID prefix and found no resolution record" is a genuinely useful answer.

🔍 The query chain is your best debugging artifact

Log every query with its filters, scores, and returned IDs. The chain reads as the agent's reasoning about the corpus, and the failure is usually obvious in it:

1. "checkout latency incident march closed without fix"   max_score 0.29  ✗
2. "checkout latency incident"                            max_score 0.71  → INC-4471
3. "INC-4471 resolution"                                  max_score 0.88  ✓

That's a healthy chain — a broad failure, a narrowing, an entity follow. An unhealthy one shows five cosmetic rephrasings of the same failed query, which points at a missing signal rather than a missing document.

✅ Worth tracking as a metric: searches per answered question. Rising means retrieval quality is dropping and the agent is compensating with volume.

The takeaway

Single-shot retrieval answers the question as worded. Make search a tool the agent uses repeatedly, return scores and an honest below-threshold signal so it knows when a search failed, expose filters as parameters, and keep exact matching in the path so entity-following works. Then bound it — a budget, query deduplication, and a stop rule that reports the search rather than inventing the answer.