Building an Eval Set From Production Traces

Hand-written eval cases test the situations you thought of. The failures that matter are the ones you didn't, and those are already sitting in your production logs. Turning them into an eval suite is a mechanical process, and it's the highest-return week of work available to most agent teams.

What to capture

You cannot build this retroactively from partial logs, so capture properly from the start:

• The complete message array sent on every model call, not just the first.
• Every tool call with its arguments, its full result, and its latency.
• The model and prompt versions in effect.
• The final output and, where you can get it, what happened next — did the user rephrase, abandon, or escalate?

That last item is the most valuable and the most often absent. A user rephrasing the same request three times is a labelled failure, free of charge, and no explicit feedback mechanism produces labels at that volume.

Selecting cases worth keeping

Don't sample randomly. A random sample is mostly easy cases you already handle, and it produces a suite that stays green while your real problems continue.

Select deliberately across four buckets:

Known failures. Anything a user complained about, abandoned, or immediately rephrased. Highest value: these are labelled by outcome.

Outliers. Runs in the top few percent by turn count, by cost, or by latency. Something unusual happened, and unusual is what you're not testing.

Escalations and errors. Every run that hit a turn limit, escalated, or ended on a tool error.

Boring representative cases. A modest set of ordinary successful runs, so you notice when something that used to work stops. Don't skip these — a suite made only of hard cases can't detect a regression on the common path.

Freezing a trace

A trace becomes a test case when it's deterministic. That means the tool responses come from the recording rather than the live system.

case:  refund_outside_window_2026_08_14
input: "I want a refund for order A-4471"
tool_responses:
  lookup_order(order_id="A-4471")  -> {recorded response}
  check_window(order_id="A-4471")  -> {recorded response}
clock: 2026-08-14T09:12:00Z

Two details that decide whether this works:

Key responses by tool and arguments, not by call order. If the agent takes a different path on replay — which is the entire point — an order-indexed recording returns the wrong data. Keyed lookup handles reordering naturally.

Decide what happens on an unrecorded call. If replay hits a call you didn't record, the honest default is to fail the case rather than pass through to the live system. A pass-through turns a deterministic test into a flaky one, silently.

⚠️ Scrub the recordings. Production traces contain customer data, and an eval fixture set is a copy of it in a repo. Redact at capture time, not later.

Writing the assertion — the part that needs judgment

The recorded run is not the specification. If it were the correct behavior you wouldn't be turning it into a test.

For each case, state what should have happened, as invariants:

assert not tool_called("issue_refund")           # window had closed
assert tool_called("check_window")               # must verify before deciding
assert mentions(output, "return window")         # tell the user why
assert turn_count <= 5

Notice this is derived from the requirement, not from the recording. The recording supplies the inputs; you supply the expectation. This is where a production trace becomes a test rather than a snapshot — and it's also where you often discover the requirement was never written down anywhere.

Keeping it useful over time

✅ One case per distinct failure mode, not per incident. Ten reports of the same underlying bug are one case. A suite that grows with incident count rather than with failure-mode count becomes slow and unloved.

✅ Retire cases whose failure mode is now impossible. If the fix was structural — a tool that can no longer be called wrongly — the case is dead weight.

✅ Re-sample quarterly. Traffic shifts. A suite built on last year's requests slowly stops representing production, and nothing announces it.

✅ Track suite coverage against real traffic. Cluster recent production runs and check that each cluster has at least one case. The gaps are where the next surprise comes from.

The takeaway

Your production logs are an eval set that already knows what your users actually send. Capture full arrays, tool results, and downstream user behavior; select failures, outliers, and a slice of the ordinary; freeze the tool responses keyed by arguments; and write the assertions from the requirement rather than from the recording. The result is a suite that fails for the reasons your users are already noticing.