How Many Eval Cases Do You Actually Need?

A team compared two prompt versions on a twenty-case eval suite. The old one scored 18/20, the new one 19/20. They shipped the new one.

They had measured nothing. At that sample size, those two results are entirely consistent with the versions being identical — and equally consistent with the new one being worse. The number moved; the system didn't necessarily.

Sample size for agent evals gets decided by how many cases someone had time to write. It should be decided by how large a difference you need to detect.

What a small suite can actually tell you

A pass rate is an estimate, and every estimate has an interval around it. For a suite scoring 90%:

n = 20    95% interval ≈ 70% – 97%
n = 100   95% interval ≈ 83% – 94%
n = 400   95% interval ≈ 87% – 93%

At twenty cases, "90%" means somewhere between mediocre and excellent. Two versions whose intervals overlap that much cannot be distinguished by their scores, however carefully you stare at them.

The intervals narrow with the square root of the sample, which is the uncomfortable part: quadrupling your cases only halves your uncertainty.

Sizing by the difference you care about

The practical question isn't "how many cases is enough" in the abstract — it's how big a change you need to catch.

Rough figures for comparing two versions, using the standard approximation for detecting a difference between two rates:

• A 20-point difference (70% vs 90%) → around 60–70 cases per version. Detectable with a modest suite.
• A 10-point difference (80% vs 90%) → roughly 200 per version.
• A 5-point difference (85% vs 90%) → roughly 700 per version.

⚠️ Most prompt changes produce differences in the low single digits. Which means most eval suites, at their actual size, cannot detect the effect of most changes being tested against them — and the score still moves, so people act on it.

The paired comparison escape hatch

Fortunately, comparing two versions doesn't require the independent-samples arithmetic above. Run both versions on the same cases and compare per case:

                    v13 pass   v14 pass
case_001               ✓          ✓      tie
case_002               ✗          ✓      v14 wins
case_003               ✓          ✗      v13 wins
...
                       ties: 78   v14 wins: 14   v13 wins: 4

Ties carry no information about which is better, so they're discarded, and the test is on the eighteen cases where the versions disagreed. This is dramatically more sensitive at the same sample size, because it removes the variance from cases both versions handle identically.

→ For agent evals this is nearly always the right design. You control the inputs, so pairing is free, and it turns a hundred-case suite into something that can detect a real change.

Different jobs need different sizes

Safety and correctness invariants — forbidden tool never called, schema always valid, approval gate always fires. These need small suites, because the expected rate is 100% and a single failure is signal rather than noise. Twenty well-chosen cases is a legitimate gate here.

Quality comparison between versions — needs the paired design, and enough disagreement cases to matter.

Absolute quality claims — "our agent handles 92% of requests" — needs the largest sample of all, plus a suite that genuinely represents production traffic. This is the claim people make most casually and support least.

💡 Stratify rather than scaling blindly

Four hundred randomly-drawn cases will be mostly easy ones, because production traffic is mostly easy. The interval on the overall number narrows while telling you nothing about the hard cases where changes actually show up.

Better: partition by the properties you care about — ambiguous requests, long contexts, tool failures, multi-intent inputs, each risky category — and size each stratum to detect a change within it. Forty cases in a stratum where a regression would matter beats four hundred spread across ground you already cover.

Report per stratum, too. An aggregate that averages a hard-case regression against easy-case stability is a number engineered to hide the thing you needed to see.

Nondeterminism multiplies everything

Each case run once is one sample from a stochastic process. Running each case several times and treating the pass rate as the measurement is more honest — and it means your effective sample budget is cases × repetitions.

Given a fixed budget, the trade is usually worth making toward repetitions on a smaller, better-chosen set of cases than a single run over many mediocre ones.

✅ What to do with the suite you have

Most teams have twenty to fifty cases and no capacity to reach seven hundred. That's workable, if you stop asking it the wrong question:

• Treat the small suite as a smoke test and invariant gate, not a quality measurement.
• Use paired comparison for version changes, and report wins/losses rather than two percentages.
• Report the interval, not just the point estimate. "90% (70–97%, n=20)" is honest and stops the one-point conversation before it starts.
• Get quality signal from production instead — completion rate, rephrase rate, escalation rate — where the sample size is your actual traffic.
• Grow the suite from real failures, so each addition covers something you've observed rather than something you imagined.

The takeaway

A suite's size determines what questions it can answer, and most suites are asked questions several times larger than they can support. Use small suites for invariants where one failure is signal. Use paired comparison for version changes, reporting wins and losses on the cases that disagreed. Stratify rather than scaling blindly, and quote intervals so nobody ships on a one-case difference again.