Retrieval Returned Zero Rows and the Agent Cited a Policy Doc Anyway

A support agent was asked whether a customer on the legacy plan could claim a refund after 45 days. It answered confidently: yes, under the 60-day window described in the refund policy, section 4. The section number was plausible. The tone was exactly right. The document did not exist.

What happened underneath is duller than it looks. The retrieval call matched nothing — wrong index, stale filter, a tenant id that didn't resolve — and returned an empty array. That empty array was serialized into the tool result as [], dropped into the conversation between a system prompt saying "answer using the retrieved policy documents" and a user question demanding an answer. The model did what the prompt asked. It had no documents, so it produced the document the prompt implied should be there.

This is not a knowledge failure. The model never claimed to know the refund window from training. It claimed to be reading a source. The bug is that nothing in the loop distinguished "the source says X" from "no source was read."

🔍 Why an empty result is the most dangerous tool output

Most tool failures are loud. A timeout throws. A 500 comes back with a status code. A malformed query raises. The agent sees a failure and usually does something reasonable — retries, or says it couldn't reach the system.

An empty result is a success. status: 200, results: []. Every layer between the vector store and the model treats it as a normal response, because it is one. By the time it reaches the model it looks identical to a successful search that happened to be narrow — and "narrow" is exactly the shape of a result a model is comfortable filling in around.

Worse, [] carries no explanation of why it's empty. These three situations produce byte-identical tool output:

• The corpus has no document about refunds. (Correct answer: "we have no policy on this.")
• The query embedded badly and matched nothing above threshold. (Correct answer: "retry with different terms.")
• The tenant filter was wrong and the agent searched an empty partition. (Correct answer: "this is broken, escalate.")

One of those means the answer is genuinely unknown. Two of them mean the agent should try again or stop. Collapsing all three into [] guarantees the model picks wrong, and its wrong pick is to keep going.

Fix one: make the empty result loud

Don't return an empty container. Return a typed signal that the model cannot read past.

if not hits:
    return {
        "status": "NO_RESULTS",
        "searched_index": index_name,
        "filters_applied": filters,
        "candidates_before_threshold": len(raw),
        "instruction": "No source documents were retrieved. "
                       "You MUST NOT answer from prior knowledge. "
                       "Either retry with a broader query or reply "
                       "that the information is not in the knowledge base."
    }

Three things are doing work here.

The status string. NO_RESULTS is a token the model actually attends to, unlike an empty bracket pair. It changes the shape of the context instead of leaving a hole in it.

The diagnostics. candidates_before_threshold: 240 versus candidates_before_threshold: 0 is the difference between "your threshold is too tight, retry" and "this partition is empty, escalate." Both the model and the human reading the trace need that distinction, and it costs a few tokens.

The instruction inside the result. Guidance in a tool result sits closer to the decision than guidance in a system prompt several thousand tokens back. When a system prompt and a fresh tool result disagree about what to do next, the tool result usually wins — so put the refusal instruction where the failure is.

⚠️ One trap: don't phrase the instruction as "apologize to the user." The agent's next move after an empty result is often a better query, not a surrender. Give it both branches explicitly.

Fix two: force citation IDs from a closed set

Loud empties stop the obvious case. They don't stop the subtler one, where retrieval returns three documents about cancellation and the model answers about refunds, citing the retrieved docs as though they said something they don't.

The structural fix is to make every claim carry an identifier the agent could not have invented:

1. Stamp every retrieved chunk with an opaque id at retrieval time — doc_7f3a91#c4, not refund-policy.md. A readable filename is guessable; a hash is not.
2. Require the answer to be structured, with each claim naming the chunk id it came from.
3. Validate the ids after generation, in code, not in the prompt. Every cited id must appear in the set actually returned by the retrieval call for this turn.
4. On a miss, do not ship the answer. Either strip the unsupported claim or fail the turn.

That validator is about ten lines and it converts an invisible failure into a caught one:

returned = {c["id"] for c in retrieved_chunks}
cited    = {c["source_id"] for c in answer["claims"]}
unsupported = cited - returned
if unsupported:
    raise UngroundedAnswer(unsupported)

The point is the direction of the check. Prompting "only cite retrieved documents" asks the model to police itself using the same context that misled it. Set membership in your own code does not care how confident the model was.

💡 The eval that would have caught it

Add a case to your suite with an empty corpus.

Take ten questions the agent answers well. Run them with retrieval hard-wired to return nothing. The only passing behavior is an explicit "not found" or a retry. Any substantive answer is a failure, including a correct one — an agent that gets the refund window right from training data with zero documents retrieved is the same bug that got it wrong last time, and it will get it wrong on the next question.

This eval is cheap, deterministic, and needs no golden answers. It also catches the regression that reintroduces this bug later, when someone widens a filter or swaps an embedding model and the empty path goes quiet again.

Grounding is not something a model can promise you in a system prompt. It's a property you enforce at the boundary: a tool result that cannot be mistaken for silence, and a citation that has to match something you actually handed it.