Eval Cases Where the Right Answer Is "I Don't Know"

Almost every eval suite consists of cases with a correct answer, and measures whether the agent produced it. That trains your attention on a single failure mode — being wrong — while leaving the more common and more damaging one unmeasured: being confidently wrong when the correct response was to stop.

An agent never tested on cases where it should decline has never been given a reason to.

Five case types worth adding

1. The information isn't there. A question requiring a fact absent from every tool and document available. Correct: say so.

What usually happens instead is the agent produces the most plausible answer from parametric knowledge, delivered with the same confidence as a retrieved fact. ⚠️ This is the highest-value case type to add, because in production this failure is silent — nothing distinguishes a retrieved answer from an invented one in the output.

2. The request is ambiguous. Two readings, materially different consequences. Correct: ask.

"Cancel the subscription for account 8821"
   → the account has two active subscriptions

Assert that a clarifying question was asked and no cancellation happened.

3. It's out of scope. A legitimate request this agent isn't for. Correct: say so and route.

4. A tool failed and the answer depends on it. Inject an error or a timeout on the tool carrying the needed data. Correct: report the failure, not a guess.

The specific failure to catch: an empty result read as a negative result. "No refunds found" because the query errored, reported as "you have no refunds."

5. The premise is false. "Why was my order cancelled?" when it wasn't. Correct: correct the premise. Agents are strongly inclined to accept premises and construct explanations, and this is the case type that exposes it.

What to assert

Not the wording — the behavior:

assert not tool_called(run, "issue_refund")     # took no action
assert asked_question(run) or escalated(run)    # used an exit
assert not contains_fabricated_value(run.output, FIELDS)

For "should have asked," the reliable assertion is that the run ended in a question or an escalation rather than a completion. That's why having an explicit escalation tool matters: it makes the correct behavior assertable, where prose is not.

The threshold problem

These cases sit on a spectrum, and both extremes are failures. An agent that never declines is dangerous; one that declines constantly is useless.

So the suite needs both halves:

• Should-decline cases where declining is correct.
• Should-proceed cases where the information is present but requires a couple of inferential steps, and declining would be over-cautious.

✅ Report them together. A change that improves one and destroys the other is a common outcome of "be more careful" prompt edits, and only the paired view catches it:

should_decline:  18/20 correct   (was 11/20)
should_proceed:  38/45 correct   (was 44/45)   ← regression

That table shows a prompt change that made the agent cautious rather than calibrated. Without the second row it looks like a clean improvement.

Sourcing the cases

The best ones come from production:

• Requests where a user immediately rephrased — usually ambiguity the agent resolved silently and wrongly.
• Outputs where a human later found a fabricated detail.
• Runs where a tool errored and the agent continued anyway.
• Support escalations where the answer given was confidently incorrect.

Each becomes a case whose expected behavior is the one the agent didn't take. → These are also the cases most likely to recur, since they represent real ambiguity in your domain rather than hypothetical ambiguity.

The prompt side

If your suite has these cases and the agent fails them, the fix is usually explicit permission plus a mechanism:

• State plainly that "I don't know" and "I need to check" are acceptable outcomes. Many system prompts imply the opposite by emphasizing helpfulness.
• Provide the tool — an escalation or clarification action, so declining is a move rather than an absence of one.
• Include one example where declining is correct. Without it, every example demonstrates confident action.

The takeaway

Add five case types where the right answer is not an answer: missing information, ambiguity, out of scope, failed tool, false premise. Assert on behavior — no action taken, a question asked, no fabricated values — and always pair them with should-proceed cases so you're measuring calibration rather than caution. An agent that has never been evaluated on knowing its limits doesn't have any.