Your Research Agent Passed the Eval Once. Run Every Case Five Times and Grade the Worst Run.

The research agent passed its eval case: "compare the refund policies of three vendors and cite each one." It went green, the change shipped, and a week later a user got an answer built on two vendors with a confident note that the third "does not publish a policy." When the same case was rerun five times, it passed three of them.

The eval was fine. What went wrong is that it ran once. One run per case measures whether the agent can do the task. Users experience whether it will, every single time they ask.

Why the same case gives different answers

A classifier gets the same input and gives roughly the same output. An agent's run is a chain of decisions, and each link can move:

• Sampling. Even at low temperature, wording shifts slightly, and a slightly different search query returns different results.
• Tool results. Search ranking, API latency, pagination and timestamps change between runs. The agent sees a different world each time.
• Compounding. A small difference at step 2 (which source got read first) changes what the agent decides at step 3, and by step 6 the two runs are doing different work.

Setting temperature to zero narrows the first source of variation. It does nothing about the second and third. The only honest way to know how often a case passes is to run it more than once.

The technique: repeat, then score two numbers

Run each case k times. Five is a reasonable starting point for most suites. Then report two numbers instead of one:

• pass@k: did any of the k runs pass? This is a capability measure. Can the agent solve this at all?
• pass^k: did all k runs pass? This is a reliability measure. Will a user get a good answer every time?

The gap between them is where the surprise lives. Say a case passes on 90% of individual runs. That sounds solid. The chance that five runs in a row all pass is about 0.9⁵, roughly 59%. A case that "usually works" fails often enough for users to notice.

def evaluate(cases, agent, k=5):
    report = []
    for case in cases:
        results = [grade(case, agent.run(case.input)) for _ in range(k)]
        passes = sum(results)
        report.append({
            "case": case.id,
            "passes": passes,
            "bucket": "solid" if passes == k
                      else "broken" if passes == 0
                      else "flaky",
            "traces": [r.trace for r in results],  # keep them all
        })
    return report

🔍 Sort the suite into three buckets

The per-case counts are more useful than any suite-wide average, because each bucket needs a different response:

• ✅ Solid (5/5). Leave it alone. It's your regression guard.
• ❌ Broken (0/5). The agent can't do this yet. That's a capability problem: a missing tool, a bad instruction, a task out of scope. You already knew about these, because single-run evals catch them too.
• ⚠️ Flaky (1–4 of 5). This is the bucket single-run evals hide. The agent has a path to success and a path to failure, and something chooses between them at random. These cases are your real to-do list.

Diagnosing a flaky case: find the first fork

Flaky cases come with a free debugging tool. You have passing traces and failing traces for the same input. Line them up and walk forward step by step until they diverge.

In the refund-policy example, the fork looked like this:

Passing run:  search("vendor C refund policy") -> 8 results, policy page ranked 2nd
              -> fetch(policy page) -> cites it

Failing run:  search("vendor C refund policy") -> 8 results, policy page ranked 9th
              (not returned; the tool capped at 8)
              -> agent concludes "no policy published"

The search tool wasn't broken and the prompt wasn't wrong. The failure depended on where the page happened to rank on that run. The fix wasn't to tune wording until the case passed once more. It was two targeted changes:

1. Tell the agent that finding nothing is not evidence of absence: before concluding "doesn't exist," try one reformulated query or go to the vendor's own site.
2. Have the search tool report when results were truncated, so "8 of many" can't be read as "all there is."

After the change, the case went 5/5. Just as important, you know why it went 5/5, which a single lucky green run would never have told you.

Make the gate compare distributions, not averages

The dangerous regression isn't a solid case going to broken. That one shows up. The dangerous one is a prompt change that leaves the suite's mean pass rate flat while quietly moving a handful of cases from 5/5 to 3/5 and a few others the other way. The average says nothing changed. Users on those cases now get a coin flip.

So gate on the buckets:

• Any case that moves solid → flaky blocks the change until it's explained.
• The flaky count shouldn't grow, even if the mean improves.
• Report pass^k beside the mean so reviewers see reliability, not just capability.

Paying for it

Five runs per case costs five times as much, and for a large suite with long agent trajectories that adds up. You don't need to repeat everything on every commit:

• Run a k=1 smoke pass on the full suite for every change. It catches things that are plainly broken.
• Run k=5 on the cases that touch the changed area, plus every case currently marked flaky.
• Run the full k=5 sweep before a release or a model swap, since that's when the distribution shifts the most.

Keep all k traces, not just the verdicts. The traces are what make a flaky case debuggable, and throwing them away turns a clear fork into a mystery.

The takeaway

A green eval run is one sample from a distribution you haven't looked at. Until each case has been run enough times to tell "always works" from "worked this time," your pass rate is measuring your luck as much as your agent. Grade the worst run, fix the forks, and treat the flaky bucket as the most honest report your suite produces.