Your 95% Step Accuracy Is a 60% Agent

A code-migration agent edits about eight files per run. The eval harness scores every individual edit against a reference, and the dashboard has been sitting comfortably around 95% for weeks. In practice, a person has to rescue roughly two runs out of three.

Both numbers are correct. They measure different things, and only one of them is the product.

The arithmetic that explains a third of it

The obvious first move is to multiply. Eight independent steps at 95% gives roughly 66% of runs completing clean — take that as illustrative rather than measured, but the shape is right. A per-step number that looks excellent is an end-to-end number that looks broken, and nothing is wrong with either measurement.

This gets quoted a lot, and it is worth knowing. It is also the least interesting part of the gap, because the independence assumption it rests on is wrong in both directions.

Failures are correlated, so runs are bimodal

If the agent misreads the codebase's convention at step one — the wrong import style, the wrong test framework, an assumption about how config is loaded — then steps two through eight all inherit it. Eight scored failures, one actual cause.

The distribution this produces does not look like the multiplication suggests. Instead of most runs landing near-perfect with an occasional slip, runs cluster at the ends: clean runs where the agent got oriented correctly, and disaster runs where it did not. A per-step average smears those two populations into one meaningless middle number.

🔍 A quick check on your own eval data: of the runs that failed, what fraction failed on three or more steps? If that fraction is high, the agent does not have a step-quality problem. It has a small number of shared root causes, each cheap to fix once found, and the averaged dashboard is actively hiding them.

The step that passes on a wrong premise

The more damaging effect is subtler. Step four reads a value that step two wrote incorrectly, then does exactly the right thing with it. Against a reference that only checks "did this step do a reasonable transformation," it scores as correct.

So the scorer records a pass on a step that propagated a fatal error. The run was already dead at step two, and the per-step metric kept reporting health for six more steps. This is why step accuracy can be improved without run success moving at all: the metric has no notion of whether the state the step operated on was valid.

What actually predicts the end-to-end number

A step being wrong is not the same as a run being ruined. A bad step has three possible fates:

• Detected and repaired — the type checker fails, the agent reads the error and fixes the edit. Costs tokens and latency, costs nothing in outcome.
• Detected and aborted — the run stops and says so. Cheap and honest; a human picks it up knowing where it stopped.
• Undetected — it flows downstream and shows up as a broken build, a subtly wrong migration, or a rescue two hours later.

Only the third fate compounds. Which means the quantity to track is not per-step accuracy at all; it is the rate of undetected bad steps.

That reframing changes what an improvement looks like. An agent at 90% step accuracy that catches most of its own mistakes will finish more runs than one at 97% that catches none, because in the first case a wrong step costs a retry and in the second it costs the run. For a code agent this is unusually tractable — compile after each edit, run the affected tests, and a large share of bad edits convert from undetected to detected before they can poison anything.

Why the misleading metric is the one you have

Per-step labels are cheap. A reference edit can be diffed automatically, the scoring parallelizes, and a thousand of them cost about what one afternoon of human review costs.

Whole-run success needs a person to look at the finished branch and say whether it was actually acceptable. That is slow and expensive, so it gets sampled rarely, if at all.

⚠️ The predictable result: the cheap metric becomes the optimization target. Prompt changes get accepted because step accuracy rose a point, while the number nobody is watching stays flat. Optimizing an average over a bimodal distribution mostly moves the population that was already succeeding.

The four numbers worth putting on the dashboard

1. Whole-run success rate, on a small sample judged by a human. Twenty runs is enough to see direction, and direction is what matters. This is the only ground truth in the list.
2. Unrecovered error rate. Of the steps that were wrong, what fraction survived into the final output? This is the number that should be driving work.
3. Failure position distribution. Where in the run do fatal errors originate? A pile-up at step one means the agent is being set up badly — missing context, ambiguous instructions — and no amount of per-step tuning will help.
4. Multi-step failure share. What fraction of failed runs failed on three or more steps? High means shared root causes; low means genuine independent variance.

Instrumenting for all four at once

Structuring the loop so each step reports its own fate produces every number above from one log:

for step in plan:
    result   = act(step)
    verdict  = verify(result)        # compile, test, schema check, assertion
    if verdict.ok:
        outcome = "pass"
    elif can_repair(verdict):
        result, outcome = repair(step, verdict), "repaired"
    else:
        outcome = "aborted"; break

    log(step_index, verdict.ok, outcome)

log_run(final_human_judgment)

Unrecovered errors are the steps logged verdict.ok == false, outcome == "pass" — the verifier missed them. Failure position is the first non-ok index. Multi-step share falls out of counting non-ok steps per failed run. All of it comes free once verify exists as a distinct thing from act.

The investment this reframes

Improving the model and improving the verifier compete for the same effort, and a per-step dashboard will always argue for the model, because that is the number it moves.

For a long task, verification usually wins. A step-level check that turns silent failures into loud ones does not need to be clever, does not need a model, and repays itself on every step of every run. The number worth watching for an eight-step agent is not how often a step is right. It is how often a wrong step gets out alive.