Assert on the Trajectory, Not the Answer

An agent eval suite that only checks final answers will pass a run that deleted a production record, took nineteen turns to do a three-turn job, and got the right answer by accident. The output matched. Everything that mattered went unexamined.

Agents differ from ordinary functions in that the path is part of the behavior. Testing only the return value throws away most of the signal, and it's the part that predicts production failures.

What a trajectory assertion looks like

Given a run that produces a trace — an ordered list of tool calls with arguments and results — assertions become straightforward:

run = agent.run("Refund order A-4471 if it's within the return window")

assert tool_called(run, "lookup_order")
assert called_before(run, "lookup_order", "issue_refund")
assert not tool_called(run, "delete_order")
assert turn_count(run) <= 6
assert arg_of(run, "issue_refund", "order_id") == "A-4471"

Five lines, and they catch classes of failure a final-answer check never sees: the agent that refunds without checking eligibility, the agent that reaches for a destructive tool, the agent that flails for nineteen turns before arriving somewhere reasonable.

The four assertions worth having everywhere

1. Forbidden tools. For each eval case, name the tools that must not be called. This is the highest-value assertion in the suite and the cheapest to write. It's also the one that catches regressions after a prompt change, because prompt edits shift tool selection in ways nobody predicts.

2. Required precondition ordering. Not the full sequence — just the dependencies that actually matter. lookup_order before issue_refund; check_permissions before send_email. Asserting a complete ordering makes the test brittle; asserting the two or three real preconditions makes it durable.

3. A turn ceiling. Pick a number a little above the reasonable path. This catches the flailing runs that a correctness check happily passes, and it's your early warning that something regressed in tool descriptions or error handling.

4. Argument-level checks on consequential calls. For any tool that writes, spends, or deletes, assert the arguments. "It called issue_refund" is not the assertion you want; "it called issue_refund with the order the user named and an amount not exceeding the order total" is.

⚠️ The brittleness trap

The obvious next step — record a known-good trace and assert the new run matches it — produces a suite that fails on every harmless variation and gets disabled within a month.

The distinction that keeps trajectory tests alive: assert invariants, not sequences. An invariant is a property that must hold for any correct run. A sequence is one correct run's particular shape.

• ❌ assert run.tools == ["lookup_order", "check_window", "issue_refund"]
• ✅ assert called_before(run, "check_window", "issue_refund")

Under the first, an agent that checks the window before looking up the order — equally correct — fails. Under the second, only genuinely wrong orderings fail.

Where the traces come from

Two sources, and you want both.

Synthetic cases are written by hand for known-risky situations: the ambiguous request, the request that shouldn't trigger any tool, the one where the right answer is to ask a clarifying question. These are cheap to write and they encode your actual requirements.

Replayed production traces are the ones that find what you didn't think of. Capture real runs, freeze the tool responses so replay is deterministic, and turn any run that went wrong into a regression case with assertions describing what should have happened. 🔍 A frozen trace is also the only honest way to compare two prompt versions — same inputs, same tool responses, only the prompt differs.

Handling the nondeterminism

Trajectory assertions are more stable than output assertions, but they aren't deterministic. Two habits keep the suite trustworthy:

• Run each case a handful of times and assert on the rate. A forbidden-tool violation should be zero out of five; a turn ceiling might reasonably be four out of five. Encoding the threshold explicitly beats pretending a single run is the truth.
• Report flips, not just failures. A case that passed consistently and now passes three times out of five hasn't failed yet, and it is the most useful signal in the report. Aggregate pass rates hide exactly this.

The takeaway

For an agent, the path is the behavior — where the money moves, where the data gets touched, where the cost accumulates. Assert what must never happen, what must happen first, how long it may take, and what arguments the dangerous calls receive. Keep the assertions to invariants so they survive harmless variation, and feed the suite from real traces so it tests failures you actually have.