Was It Retrieval or Reasoning? Diagnosing a Wrong Answer
A day spent rewriting the prompt for a wrong policy answer, when the policy document was never retrieved. Three checks in the right order, and one log field that separates "ignored it" from "never saw it."
A support agent told a customer the refund window was fourteen days. The policy document says thirty. The team spent most of a day rewriting the prompt to be more careful about policy questions.
The policy document was never retrieved. No prompt would have helped, because the answer was never in the context to be read.
Why the prompt gets blamed first
The prompt is the part you can edit in ten seconds. Retrieval feels like infrastructure — a scored list produced by a component someone else configured. So the instinct is to treat a wrong answer as a reasoning problem and start rewording.
The check that avoids the wasted day takes five minutes, and the order matters more than the steps.
The three-step check, in this order
1. Is the answer in the corpus at all?
Search manually for the fact. If the current policy exists nowhere, or exists only in a document that was superseded and never removed, this is a content problem. Stop — there is nothing to fix in the agent.
⚠️ This step catches more than people expect, and skipping it is how teams spend a week tuning retrieval over a corpus that doesn't contain the answer.
2. Was it retrieved?
Look at what the retrieval call actually returned for that specific run — the document IDs and their scores. If the right document isn't in that list, it's a retrieval problem and the prompt is irrelevant.
3. Was it retrieved and still not used?
If the right document was returned and survived into the final context, only now is it a reasoning question — or a position problem, or a conflict problem.
→ Most investigations should terminate at step 1 or 2. Almost nobody runs them in this order.
The four retrieval failure modes
Once you're at step 2, the scores tell you which one:
Vocabulary mismatch. The question says "money back," the document says "refund eligibility period." Max score low, nothing relevant returned. → Fix with hybrid keyword-plus-vector retrieval, or by adding the customer-facing phrasing to the document.
Chunk boundary. The document contains the answer; the chunk doesn't, because "thirty days" sits in a table two chunks away from the heading that gives it meaning. The document scores well and the returned chunk is useless. → Fix by chunking on structure and prepending section context.
Ranked below the cutoff. The right chunk came back at position nine with k=5. Score respectable, just not top-five. → Fix by raising k, reranking, or filtering more aggressively before ranking.
Filtered out by metadata. Excluded by a product filter, a locale, or an archived flag that was set wrongly. Score irrelevant — it never entered the candidate set. → The sneakiest one, and the reason to log filters alongside queries.
💡 The conflicting-sources case
Worth separating because it looks exactly like a reasoning failure and isn't.
The right document was retrieved. So was an old one saying fourteen days. Both are in the context, both look authoritative, and the agent picked one. That's not the model reasoning badly — it's being asked to adjudicate a contradiction in your corpus with no basis for choosing.
✅ Before concluding "the model got it wrong," check whether anything else in the retrieved set contradicts the right answer. If so, the fix is corpus hygiene — supersede the old document, or add dates and a clear source-of-record so precedence is decidable.
🔍 The log field everyone omits
Most retrieval logging records the query and the returned IDs. The field that closes the gap between step 2 and step 3:
retrieval q="refund window" filters={product: "pro"}
returned: [doc_88 (0.81), doc_12 (0.77), doc_44 (0.71),
doc_09 (0.66), doc_31 (0.62)]
in_context: [doc_88, doc_12, doc_44] ← truncated at budget
dropped: [doc_09, doc_31] reason: retrieval cap
Retrieved is not the same as present. A document can rank well, be returned, and then get cut by the context budget before the model ever sees it. Without that line, step 2 says "it was retrieved" and step 3 says "the model ignored it," and you go looking for a reasoning failure that doesn't exist.
Fixes, by where you landed
| Diagnosis | Fix |
|---|---|
| Not in corpus | Content — write it, or remove the stale version |
| Vocabulary mismatch | Hybrid retrieval; add real user phrasing to docs |
| Chunk boundary | Chunk on structure; prepend section context |
| Below cutoff | Raise k, add reranking, filter before ranking |
| Filtered out | Audit the metadata and the filter logic |
| Retrieved, dropped by budget | Raise the retrieval allocation or shrink what else competes |
| Contradicted by another source | Corpus hygiene — supersede, date, name a source of record |
| Present, correct, unused | Now it's the prompt, position, or the model |
Only the last row is a prompt problem. It's also the row people start on.
The takeaway
Check whether the fact exists, then whether it was retrieved, then whether it survived into the context — in that order, before touching the prompt. Log the queries, the filters, the scores, and crucially which returned documents actually made it into the final array. Most wrong answers that look like bad reasoning are a missing document, a bad chunk boundary, a metadata filter, or a contradiction you shipped in the corpus — and none of those care how the instruction is worded.