Green Evals, Broken Production: The Frozen Fixture Problem

A warehouse-reconciliation agent passes all forty-eight eval cases on Tuesday afternoon. On Wednesday morning it silently writes a day of revenue figures into the wrong currency column, and nobody notices until a dashboard looks off on Thursday. Nothing about the agent changed between Tuesday and Wednesday. Nothing about the evals changed either — that's the problem.

The evals were built from recorded tool responses captured months ago. Since then the upstream billing API added an amount_minor field and started returning currency as an ISO code instead of a symbol. The agent's live prompt now sees a payload it has never been evaluated against. The suite is still green because it is still grading the agent against a world that stopped existing in March.

What frozen fixtures actually break

Recording tool calls is the right instinct. Live calls in an eval suite are slow, cost money, hit rate limits, and make results non-reproducible — a flaky third party turns your regression signal into noise. So teams snapshot the responses and replay them. Reasonable.

The trap is what the snapshot silently becomes. On day one, a fixture is a faithful sample of reality. By month six it is an assertion — an unexamined claim that the tool still behaves this way. Nobody ever wrote that claim down, so nobody ever reviews it, and it never fails. Fixtures do not rot loudly; they rot into false confidence.

The damage is specific and worth naming:

• Shape drift. Fields added, renamed, nested, or made nullable. The agent's parsing logic is only exercised against the old shape.
• Semantic drift. Same field, new meaning. status: "complete" used to mean settled; now it means accepted-for-processing. The schema validates perfectly and the agent draws the wrong conclusion.
• Distribution drift. The recorded response is a clean, small, well-formed one. Production now returns 4,000 rows with nulls scattered through them, and the agent's summarization step quietly truncates.
• Error drift. The tool used to return 429 on overload; it now returns 200 with an error object in the body. Your recorded happy path never touches the new failure branch.

Shape drift is the friendly one — it usually crashes. Semantic drift is the expensive one, because both the fixture and the live response parse cleanly and only one of them is true.

🔍 Three cheap detectors

None of these require rebuilding your eval harness. All three run outside the eval loop, so they cost nothing per eval run.

1. Contract-test the live tool against the recording

Once a day, call each tool for real with the same arguments the fixture recorded, and compare the structure of the response — not the values. Values legitimately change; structure changing is news.

def check_fixture(fixture):
    live = call_tool(fixture.tool, fixture.args)
    live_shape  = shape_of(live)          # keys + types, recursive
    saved_shape = shape_of(fixture.response)
    if live_shape != saved_shape:
        report_drift(fixture.id, diff(saved_shape, live_shape))

This is a handful of calls a day and it catches shape and error drift before an eval run ever lies to you. Run it as its own scheduled job, and let it fail loudly and separately from the eval suite — a drift report is a different signal from a regression, and merging the two teaches people to ignore both.

2. Store a schema hash with every fixture

Hash the normalized key-and-type structure of each recorded response and store it in the fixture file. Then any code path that loads a fixture can compare that hash against the tool's current published schema, if the tool has one, or against the last contract-test result.

{
  "tool": "billing.list_invoices",
  "captured_at": "2026-03-04",
  "schema_hash": "sha256:9f1c...",
  "response": { "...": "..." }
}

The hash turns "is this fixture current?" from a judgment call into a comparison. It also gives you a grep-able inventory: one command tells you every eval case still pinned to a schema version the tool no longer serves.

3. Give fixtures an expiry, not just a birthday

captured_at is passive information nobody reads. A max age is a rule the harness can enforce. Pick a window per tool based on how fast that tool moves — an internal service your own team ships weekly might get 30 days; a stable public API might get 180 — and have the harness emit a warning when a fixture crosses it and fail the run when it crosses double.

⚠️ Expiry deliberately produces work you did not ask for. That is the point. A fixture that nobody has re-recorded in a year is not stable, it is unowned.

When a case should call the live tool

Recorded-by-default is right. The useful question is which cases earn an exception.

Record when:

• ✅ The case tests the agent's reasoning given a known input — planning, tool selection, how it handles a specific payload.
• ✅ You need determinism to detect a prompt or model regression.
• ✅ The call is expensive, rate-limited, or has side effects.

Go live when:

• ✅ The case tests integration — auth, pagination, timeouts, the actual serialization of a real call.
• ✅ The tool is one your own team owns and changes frequently.
• ✅ The case covers a path where semantic drift would be silent and costly, like a financial field or a permissions check.

A practical split: a large recorded suite that runs on every commit, plus a small live suite — five to ten cases — that runs nightly against a sandbox account and is allowed to be a bit flaky. The recorded suite tells you the agent changed. The live suite tells you the world changed. Conflating them is why teams end up with one number that means neither.

The takeaway

An eval fixture is not test data; it is a dated claim about an external system, and it needs the same maintenance any other dependency gets. Treat a recording without an expiry date the way you'd treat a pinned dependency with no upgrade path — it is not stability, it is deferred breakage. Contract-check the live tools on a schedule, hash the shapes you depend on, and keep a handful of live cases running so something in your pipeline is still talking to the real world.