Your Eval Ran Each Case Once. That's Why It Missed the Flake.

An inbox-triage agent has a forty-case eval suite. Every case passes. It ships, and within a week someone reports that a contract renewal got filed under "newsletters." You pull up that exact case from the suite and rerun it. It passes. You run it four more times. It fails twice.

Nothing regressed. That case was never a pass. It was a 60% pass that you sampled once and wrote down as green.

A green case is a sample, not a measurement

An agent is a stochastic policy wrapped around a stochastic model. Sampling temperature moves the first token. Tool latency changes what comes back within a timeout. A retriever returns a slightly different neighbour set because an index was rebuilt. The model picks search_threads before read_thread on one run and after it on the next. Running a case a single time draws one sample from that distribution and records it as a fact.

The arithmetic bites in a specific way. Say a suite has 40 cases: 34 that genuinely pass essentially always, and 6 that pass roughly 70% of the time. Run each once and the chance all six shaky ones land green together is 0.7⁶ — a little over one run in ten. Most runs come back red, so the suite gets a reputation for being flaky and people start rerunning until it's green.

Now flip it. Suppose those six cases pass 95% of the time instead. The whole suite comes back fully green about three runs in four, and the 5% failure mode never appears on anyone's screen — until it appears on a user's.

Those look like opposite problems. They're the same missing number: nobody knows the per-case pass rate, so nobody can tell a real regression from a resample.

⚠️ The dangerous case isn't the one that fails. It's the one that fails rarely enough to look like noise and often enough to reach a user.

Run it k times and report two numbers

The change is small and the cost is real:

def evaluate(case, k=5):
    outcomes = [run_agent(case) for _ in range(k)]
    ok = [case.assert_ok(o) for o in outcomes]
    return {
        "case": case.id,
        "pass_rate": sum(ok) / k,        # how often it works
        "consistent": all(ok),           # did it work every single time
        "failures": [o for o, good in zip(outcomes, ok) if not good],
    }

Two numbers per case, and both belong in the report:

• Pass rate — the estimated probability the case works. This is the number you track over time.
• Consistency (all-of-k) — did every run pass. This is the number you gate a release on for critical paths.

Suite-wide, publish the mean pass rate and the count of cases that were consistent. "38/40 cases green" tells you almost nothing. "36/40 consistent at k=5, mean pass rate 0.94" tells you where to look.

Be honest about what k buys. With k=5, a case that truly passes 60% of the time sweeps clean only about 8% of the time — you'll catch it almost immediately. A case that passes 95% of the time sweeps about 77% of the time, so k=5 mostly won't find it. Resolution scales with how bad the flake is: coarse flakes are cheap to detect, fine ones are expensive. Don't pretend a k=5 sweep proves determinism.

💡 Where the repeat budget goes

Running everything at k=10 on every commit is not affordable and not necessary. Tier it:

Say a run costs roughly a few cents in tokens. At k=5 across 40 cases you're paying for 200 runs a night instead of 40. That multiple is not overhead — it's the price of the number you were previously guessing at.

🔍 Localize the flake to a step

Once a case has a pass rate below 1.0, the k runs you already paid for are a diagnostic set. Record every step of every run, then compute a pass rate per step rather than per case.

For the triage agent, that looked like:

• search_threads returned the correct thread → 10/10
• classify_intent produced the right label → 6/10
• apply_label executed correctly given its input → 10/10

The trajectory isn't flaky. One decision is. That reframes the fix from "make the agent more reliable" — which nobody knows how to action — to "constrain one classification call": tighten the label enum in the tool schema, add the two confusable categories as few-shot examples, or make the step return its top choice with an abstain option that routes to a human. Step-level pass rates turn a vague reliability complaint into a specific edit.

When 60% is the assertion's fault

Before blaming the model, diff the failing runs against the passing ones. Three assertion bugs show up constantly, and each one manufactures fake flakiness:

• ❌ Exact-string matching a free-text summary, so any paraphrase fails.
• ❌ Asserting tool-call order when order doesn't affect the outcome.
• ❌ Asserting on a field no downstream system actually reads.

If the "failures" are semantically correct and differ only in wording or sequence, the test is wrong and the agent is fine. Fix the assertion — check semantic equivalence, or assert on the final state rather than the path taken.

The trap is doing this reflexively. A loosened assertion turns a 60% case into a 100% case without changing a single thing about the agent's behavior, which is exactly what a green dashboard rewards. Loosen only when you've read the failing traces and confirmed the output was genuinely acceptable.

The takeaway

Coverage isn't the missing number in most agent eval suites. Repetition is. A pass rate with no k beside it is a coin flip reported as a fact — and the cases sitting between 85% and 99% are the ones that will find your users before your suite does.