How to Benchmark Models on Your Own Task

Public benchmarks tell you about a model. They don't tell you whether it's better for you, because your task isn't in them and the properties that make your task hard aren't what they measure. The substitute is a benchmark of your own, and it's a smaller job than it sounds.

What you're actually measuring

Not "which model is better." Four questions, each with a different answer:

• Does it complete our tasks correctly?
• How many turns does it take?
• What does it cost per completed task?
• Does it still respect our safety invariants?

⚠️ Teams that measure only the first make migration decisions that surprise them on the invoice — a model that scores two points higher and costs seventy percent more per task is a business decision, not an obvious upgrade.

Building it

Start from real inputs. Fifty to a hundred requests drawn from actual traffic, selected deliberately rather than sampled randomly: your known failures, your outliers by turn count or cost, your escalations, and a slice of ordinary successful cases so you'd notice a regression on the common path.

Freeze the tool responses. Record real responses once and replay them, keyed by tool name and arguments — not by call order, since the whole point is that a different model may take a different path. Without this you're measuring the world's variance, not the model's.

Fix the clock and any generated IDs. Injected, not ambient.

Write assertions from the requirement, not the recording. The recorded run is the input; what should have happened is your specification. This is where a captured trace becomes a test rather than a snapshot.

Running it

Multiple runs per case. A single run is one sample from a stochastic process. Five runs per case and report the rate — the numbers mean something different and you'll stop over-reading single failures.

Both models, identical everything else. Same recorded responses, same clock, same cases.

Prompts written per model. ✅ The most common way this experiment produces a wrong answer: running the incumbent's prompt against the challenger. Carry the workaround-annotated prompt as one variant and a cleaned-up version as another — a newer model often does better with a shorter prompt, because the compensations were for the old one.

Reading it

Report all four dimensions together:

                       current    candidate
pass rate               0.89       0.92
turns/task (p50)         5          7
cost/task (p50)        $0.031     $0.058
tool-selection diffs      —      11/60 cases chose differently
invariants              12/12      12/12
refusals                  2          5

Read the diverging cases individually. Eleven is a readable number. Some will be improvements; some will reveal your tool descriptions were ambiguous all along and the old model happened to guess your way.

Treat invariants as pass/fail gates, not as a score. Anything less than perfect blocks regardless of the rest.

Check the refusal column. Movement in either direction matters and rarely appears in anyone's benchmark.

💡 The part that makes it worth the effort

The suite outlives the comparison. Once it exists you have:

• A regression gate for prompt changes.
• A measurement for every future model release.
• The ability to answer "should we route this step to a cheaper tier?" with data.
• A specification of what your agent is supposed to do, which many teams don't otherwise have written down.

That last one is often the surprise. Writing assertions forces the requirements to be stated, and the exercise usually finds two or three behaviors nobody had agreed on.

🔍 The shortcut when you have no time

If a hundred cases isn't happening this week: take your ten most important real tasks, freeze their tool responses, write assertions for each, and run five times per model.

Fifty runs per model. Not statistically strong, and vastly better than impressions — and it's a couple of hours. The suite grows from there, one production failure at a time.

The takeaway

Public benchmarks measure the model; yours measures your system. Build it from real inputs with frozen tool responses, write assertions from the requirement rather than the recording, run each case several times, and report pass rate alongside turns, cost, and invariants. Read the cases where the two models diverged — that's where you learn what your tool surface was quietly relying on.