Test the Tool or Test the Agent? What Each Layer Catches

A data pipeline team had an eval suite that took eleven minutes, cost real money to run, and failed intermittently. When they sorted a month of its failures by root cause, most traced to tool contracts — a truncation flag that wasn't set, an error string that said nothing actionable, a date field that was ambiguous about timezone.

Every one of those is findable by an ordinary unit test, in milliseconds, with no model involved. They were being caught by the slowest, most expensive, least specific instrument available.

Two layers with very different economics

Tool-layer tests are ordinary unit tests. Call the handler, assert on what comes back. Deterministic, milliseconds, free, and a failure names the function.

Agent-layer evals run the loop with a model. Nondeterministic, seconds to minutes, metered, and a failure says "the run did the wrong thing" — which is a starting point for an investigation rather than a diagnosis.

Both are necessary. The question is which properties belong where, and the default is to push too much upward.

What only tool tests catch — cheaply

These are properties of your code, not of the agent's behavior:

• Empty vs failed vs filtered. Does a no-match return differ from a timeout return differ from a permissions exclusion? Three assertions.
• Truncation flags. Given more rows than the limit, is truncated true and total populated?
• Actionable errors. Does the error string say what happened and whether to retry? Assert on its structure, not its prose.
• Units and formats. Currency stated, timestamps in ISO with a zone, amounts in the documented denomination.
• Idempotency. Call the write handler twice with the same key; assert one effect.
• Permission enforcement. A negative test whose expected result is a permission error.
• Pagination completeness. Does the bounded "fetch all" variant actually stop at its cap and report it?

⚠️ Every one of these is a common cause of agent misbehavior, and none of them requires a model to detect.

What only agent evals catch

Behavior, which is what the layer is for:

• Tool selection — did it pick the right one from the offered set?
• Argument construction from ambiguous input.
• Sequencing — did it check eligibility before acting?
• Termination — did it stop when done, and not before?
• Recovery — given a well-formed error, does it route around the failure or hammer it?
• Declining — on a case where the right answer is a question.
• Injection resistance.

Notice that "recovery" appears here while "actionable error string" appears above. That's the seam: the tool test proves the message is well-formed; the agent eval proves the agent does something sensible with it. Testing both at the agent layer means you can't tell which half broke.

🔍 The diagnostic

Take your last twenty eval failures and label each by root cause:

tool returned an ambiguous empty result        6
error string gave the agent nothing to act on  4
truncation not signalled                       3
agent picked the wrong tool                    3
agent stopped early                            2
agent misread an unambiguous result            2

The first three categories — thirteen of twenty here — are tool contract bugs. Move them down a layer and the agent suite gets faster, cheaper, more stable, and more informative, because what remains is genuinely about behavior.

→ The general rule: if a failure would still be a failure with no model in the loop, it belongs in a unit test.

The split, as a table

What makes this possible

A dispatch seam. If every tool call routes through one function you own, the handlers are directly callable in a test, and the agent suite can replay recorded responses instead of hitting live services.

Without that seam, both layers collapse into one slow, flaky thing — which is how suites end up at eleven minutes and get run less and less often until nobody trusts them.

✅ The practical arrangement

• Tool tests on every commit. Fast, free, and they should be the bulk of your assertions by count.
• Agent evals on prompt, tool-description, and model changes, with recorded tool responses so they measure behavior rather than the world.
• A small live smoke test before shipping, to catch drift between your recordings and reality.

The takeaway

Most agent failures that look like behavior problems are contract problems: an ambiguous empty result, an unhelpful error, a missing truncation flag. Those are ordinary code bugs and belong in ordinary unit tests, where they run in milliseconds and name the broken function. Reserve the expensive, nondeterministic agent layer for what only it can measure — selection, sequencing, recovery, termination, and knowing when to ask. Sort your last twenty failures by cause; the split usually makes itself obvious.