How to Run an Agent Pilot That Produces a Decision

Most agent pilots end without a decision. They demonstrate that something is possible, everyone agrees it's interesting, and then nothing happens — because the pilot was never designed to answer a question that would change anyone's plans.

Designing it to produce a decision is a different exercise from designing it to produce a demo, and it isn't harder.

Start from the decision

Before anything is built, write down: what decision will this pilot inform, and what result points which way?

Decision:  do we route tier-1 support tickets through an agent?
Go if:     ≥60% of tier-1 tickets resolved without escalation,
           with customer satisfaction not lower than the human baseline,
           at under X per ticket
No-go if:  below 40% resolution, or any unrecoverable customer-facing error
Unclear:   between 40 and 60 → extend with a specific change

⚠️ Writing this before building is what prevents the most common failure: a pilot that produces impressions, followed by an argument in which everyone's prior wins.

Pick a scope that can produce a real signal

Real inputs, not curated ones. Actual tickets, actual documents, actual requests, including the ugly ones. A pilot on hand-picked inputs measures nothing about production.

A bounded slice, chosen for measurability. One ticket category, one document type, one workflow. Narrow enough to finish and to instrument.

Enough volume for the numbers to mean something. A pilot on twenty cases can't distinguish a 60% resolution rate from a 40% one. Size against the decision threshold, or accept that the result will be inconclusive by construction.

Something with a comparison baseline. What's the current resolution rate, cost, and time? Without a baseline, any result is uninterpretable.

Instrument before you start

Not after. What you'll need:

• Every run, with inputs, outputs, and trace.
• Outcome per case — resolved, escalated, wrong, abandoned.
• Cost per completed task.
• Time per case, compared to baseline.
• What a human would have done, where you can capture it. ✅ This is the most valuable field and the most often missing — it turns "the agent did X" into "the agent did X where a human would have done Y," which is the actual comparison.

Run it in shadow first

Where possible, run the agent alongside the existing process without using its output. You get the comparison with none of the risk, and you find the failure modes before anyone is affected.

Then a limited live phase, with the gate and the rollback rehearsed.

💡 Read the failures, not the average

The aggregate number decides go or no-go. The failures tell you whether the no-go is fixable.

Sort them: was this a capability limit, a missing-context problem, a specification problem, a tool problem, or a blast-radius problem? Only the first is a reason to wait. The others are reasons to fix something and re-run — and a pilot that ends "no, because the model isn't good enough" when the real cause was that nobody documented the business rules has produced a wrong decision.

What makes pilots fail to decide

• No threshold written in advance, so the result is interpreted rather than read.
• Curated inputs, so it doesn't generalize.
• No baseline, so there's nothing to compare against.
• Too small to distinguish outcomes.
• No instrumentation, so afterwards nobody can say what happened.
• Scope that grew during the pilot, so it never finished.
• Success criteria that are qualitative — "does it feel useful."

The takeaway

Write the decision and the thresholds before you build. Use real inputs on a bounded slice with enough volume to distinguish the outcomes, against a measured baseline. Instrument from the start, including what a human would have done. Run in shadow before running live. Then read the failures by cause — because the difference between "the model can't" and "we never wrote down the rules" determines whether no-go means stop or means fix one thing and try again.