From Prompt Edit to Signal in Under a Minute

Change one line in a prompt, wait six minutes to find out whether it helped. That loop, more than any framework choice, determines how good an agent gets — because the number of experiments you run per day is capped by how long each one takes.

Six minutes means roughly ten experiments in a working day, and only if nothing else needs attention. Thirty seconds means a hundred. The second team will find things the first never gets to.

What makes the loop slow

Four causes, in roughly the order they cost you time:

Live tool calls. The agent hits real services on every iteration. Slow, rate-limited, nondeterministic, and — for anything that writes — either dangerous or requiring cleanup. Also means two runs of the same prompt differ for reasons that have nothing to do with the prompt.

Running everything to learn one thing. The whole suite executes when the change affects one case. Fifty cases at a few seconds each is minutes of waiting to answer a question about case forty-two.

Deploying to change a prompt. If prompts are baked into an image, every experiment includes a build. This is the one that quietly makes people stop experimenting.

Reading traces by hand. The run finishes and you scroll through a transcript trying to spot what differs from last time. Slow, and easy to miss the thing that changed.

Fix 1: the replay seam

The foundational change, and the one everything else builds on. Route every tool call through a single function, then make it swappable:

def dispatch(call, mode):
    if mode == "replay":
        return recorded[key(call.name, call.args)]   # keyed by name+args
    return real_tools[call.name](**call.args)

Record real runs once, replay them forever. Iterations become instant and deterministic, so a behavior change between two runs is attributable to your edit rather than to the world.

⚠️ Key recordings by tool name and arguments, not by call order. The whole point is that the agent may take a different path after your change — order-indexed recordings return the wrong data exactly when the experiment gets interesting.

Decide too what happens on an unrecorded call. Failing loudly is right for the eval suite; during exploration, falling through to the live tool and recording the result is more useful.

Fix 2: a single-case runner

One command, one case, immediate output:

$ agent-run case_042 --prompt v15

case_042  "extract line items from this scanned invoice"
  turn 1  extract_fields(doc="inv_88.pdf")         1.2s (replayed)
  turn 2  validate_totals(...)                     0.1s (replayed)
  turn 3  finish(items=7)
  
  ✓ schema_valid        ✓ totals_reconcile
  ✗ currency_detected   expected "EUR", got null
  
  3 turns · 4,180 tokens · 2.4s

Everything needed to judge the edit, in seconds. The assertions matter as much as the trace — without them you're reading output and forming an impression, which is how people convince themselves a change helped.

Fix 3: load prompts from disk at run time

Prompts read from files at startup, with the path configurable. No build, no deploy, no restart if you can manage a watch. This is a small change that removes the largest fixed cost from every iteration.

The same mechanism gives you --prompt v15 for free, which makes the next fix possible.

Fix 4: diff two runs, don't read two traces

The comparison you actually want is what changed:

$ agent-diff case_042 --prompt v14 v15

trajectory:
    v14  extract_fields → validate_totals → finish
    v15  extract_fields → validate_totals → finish        same

assertions:
    currency_detected   v14 ✗ → v15 ✓                     FIXED
    totals_reconcile    v14 ✓ → v15 ✓
    
cost:   4,020 → 4,180 tokens  (+4%)
turns:  3 → 3

Ten seconds to read instead of two transcripts to compare. Run it across the whole suite and the same view shows which cases flipped in each direction — including the ones that regressed, which is the half people miss when reading impressions.

🔍 Measure the loop itself

Time it once, honestly: from saving a prompt edit to having a signal you'd act on. Most teams are surprised, because the wait is distributed across four small delays nobody counts individually.

Then keep it staged, so each question is answered at the cheapest tier that can answer it:

• Seconds — one case, replayed. The inner loop, where most work happens.
• A minute or two — the full suite, replayed, with a diff view.
• Longer, less often — live tools, a smoke test, then a canary.

✅ Don't collapse the tiers. Live runs belong at the end, not in the loop where you're iterating on wording.

⚠️ The way this goes wrong

Making the loop fast by making it unrepresentative. If replayed responses drift from what the real tools now return, the fast loop measures a fossil, and you'll tune against conditions that no longer exist.

Guard by re-recording periodically and running the live smoke test before anything ships. The fast loop is for iteration; the slow one is what you believe.

The takeaway

Route tool calls through one swappable seam and record them. Add a single-case runner that prints the trajectory and the assertions. Load prompts from disk so no build stands between an edit and a run. Then diff two versions rather than reading two traces. That's an afternoon of work, and it changes how many experiments you can afford to be wrong about — which is the thing that actually determines where the agent ends up.