Bisecting a Bad Run: Finding the Change That Broke It

An agent that worked last week doesn't work today. Nothing obvious changed. There are, however, six things that could have changed — the prompt, a tool description, a tool's return shape, the retrieval index, the model version, and the data itself.

Guessing costs days. Bisecting costs an afternoon, and the technique is a direct application of what you'd do with a code regression, adapted for the fact that an agent's inputs are assembled from six independent sources.

The prerequisite: reproduce it deterministically

Nothing works until the failure is reproducible. That means a frozen case:

• The exact input.
• Recorded tool responses, keyed by tool and arguments.
• A fixed clock.
• A pinned model version.

If the failure doesn't reproduce at least most of the time under these conditions, you have a variance problem rather than a regression — run it ten times and check whether it's a rate change rather than a break. ⚠️ Bisecting an intermittent failure produces confident wrong conclusions, because a passing run in the middle of the search sends you down the wrong half.

Bisect the dimensions, not the timeline

Unlike code, the change isn't necessarily in one commit history. Six dimensions varied independently, so test them independently.

Build the current failing configuration, then swap in the old version of one dimension at a time:

current (fails):
  prompt v18, tools v7, index 2026-08-24, model 2026-06, data current

test A: prompt v14 (last known good), everything else current  -> ?
test B: tools v5,  everything else current                     -> ?
test C: index snapshot from last week, everything else current -> ?
test D: model pinned to the older version, everything else     -> ?

Whichever swap restores correct behavior identifies the dimension. Six runs, and usually far fewer, because the likely suspects are obvious once listed. This is why versioning every dimension matters — a dimension you can't roll back is one you can't test.

Then bisect within the dimension

Say prompt v14 fixes it and the current version is v18. Four versions to search, so test v16: if it fails, the break is v15 or v16; if it passes, it's v17 or v18. One or two more runs isolates the exact edit.

Then read that diff. It's usually one sentence, and it's usually a sentence that looks harmless — a clarification added for a different case, which changed how the model weighed something else.

When the dimension is the context itself

The hardest case: nothing versioned changed, but the assembled context differs because the data grew. A conversation that now exceeds a threshold and triggers summarization. A tool result that got bigger and pushed something past the truncation limit. Retrieval returning a different set because the index gained documents.

The technique here is a diff of the raw arrays between a working and a failing run:

diff <(jq -r '.messages[].content' good.json) \
     <(jq -r '.messages[].content' bad.json)

What you're looking for: something present in one and absent in the other. → In practice the answer is nearly always that a constraint, an identifier, or a required fact is missing from the failing array, and the cause is a size threshold that got crossed. This is why the raw array is worth logging in full.

The order to test in

Prior probability, roughly:

1. The prompt. Most frequently changed, most nonlocal effects.
2. Tool descriptions and schemas. Second most edited, directly drives selection.
3. Context assembly thresholds. Truncation, summarization, retrieval count — these change behavior without anyone editing anything.
4. Tool return shape. A field renamed or removed upstream.
5. The retrieval index. New documents outranking the ones that used to be returned.
6. The model version. Rare if pinned; if you're on a floating alias, promote this to first.

✅ What makes this possible at all

Every dimension needs to be versioned and rollable-back independently. If any is unversioned, it's untestable and you're back to guessing:

• Prompts with hashes of the rendered string.
• Tool definitions with versions.
• Retrieval index snapshots, or at least a timestamp filter.
• Pinned model versions.
• Recorded tool responses per run.
• Full raw arrays retained for a window.

That list is the real investment. The bisection itself is easy once you have it, and impossible without it.

The takeaway

Six dimensions vary independently, so bisect across them before bisecting within one. Reproduce deterministically first — an intermittent failure will mislead the search. When nothing versioned changed, diff the raw arrays between a good and a bad run and look for what's missing rather than what's different. And version everything now, because that's the part you can't do retroactively when the regression arrives.