A 5% Canary Mostly Samples Your Easy Tasks. Roll Agent Upgrades Out by Task Class.

You move 5% of traffic onto the new model version. The dashboards stay green for four days. You promote to 100%, and within an hour the failures arrive: refunds above the auto-approve limit, tickets that needed three tool calls and a lookup in the billing system, the long conversations where the customer changed their mind halfway through.

The canary did not miss those cases because it was too small. It missed them because random sampling of traffic is sampling of volume, and an agent's risk does not live in its volume. It lives in a thin tail of tasks that are long, tool-heavy, and irreversible. A 5% random slice of a support queue is mostly password resets.

⚙️ Why percentage canaries under-sample agent risk

A stateless web service has roughly uniform risk per request. If the new build breaks the checkout handler, 5% of checkout traffic is enough to see it, because checkout traffic is proportional to its share of the whole.

Agent traffic is not shaped like that. Consider an illustrative support queue where most tickets resolve in one or two turns with a single read-only lookup, a smaller portion need several tool calls, and a small remainder touch money or account state. The one-turn tickets dominate the count and contribute almost none of the blast radius. The money-touching tickets are rare enough that a random 5% slice over a few days might contain a handful, none of which happened to hit the specific behavior that changed.

Worse, the easy tasks are exactly the ones a model upgrade never breaks. Anything strong enough to ship handles a password reset. What shifts between versions is judgment under ambiguity, willingness to ask before acting, how many steps it takes before it gives up, and whether it reads a tool result carefully or skims it. Those only show up where the task is hard.

Derive the task classes from your traces, not from a whiteboard

You already have the data. Pull a few weeks of production traces and bucket each run on four axes:

• Step count. How many model turns before it finished. One-shot, short chain, long chain.
• Tool surface. Which tools it actually called, not which it had access to.
• Write or read. Did it change anything outside its own transcript.
• Reversibility. If the change was wrong, can it be undone by another automated call, or does it need a human, a refund, or a support escalation.

Cluster those into four or five named classes. For a support agent, that might land as: lookup (read-only, one or two turns), guided change (writes, reversible, short), investigation (long chain, many reads, ends in a summary for a human), value action (money or entitlement changes, irreversible without a human).

The naming matters more than the taxonomy being perfect. You need classes your rollout can route on and your dashboards can group by.

Roll out along the reversibility ladder

Instead of one percentage that ratchets upward, run a ladder where each rung is a whole class, at full volume, for a fixed soak period.

rung 1  lookup          100% of class, soak 24h
rung 2  investigation   100% of class, soak 48h   # long chains, still read-only
rung 3  guided change   100% of class, soak 48h   # writes, reversible
rung 4  value action    50% of class, soak 72h    # money; keep a live control group

Routing is a lookup at dispatch, not a random number:

def pick_agent_version(task):
    cls = classify(task)          # same classifier your dashboards group by
    if cls in ROLLED_OUT:
        return NEW_VERSION
    if cls == CANARY_CLASS:
        return NEW_VERSION if hash(task.id) % 2 == 0 else OLD_VERSION
    return OLD_VERSION

Two things this buys you that a percentage never does. First, every rung gives you a complete sample of a class, so a rare behavior inside that class is not a coin flip anymore. Second, when something breaks, the rollback is scoped to a class rather than to a traffic slice, and you know which capability regressed.

The cost is real: this takes longer than sliding a dial from 5 to 100 in an afternoon, and it needs a classifier at dispatch time that agrees with the one in your analytics. Budget for that before you promise a rollout date.

Watch per-class metrics, not one aggregate

An aggregate success rate over mixed classes is dominated by the easy class and will stay green through a serious regression. Break every signal out by class, and add the ones specific to agents:

• Steps to completion, as a distribution rather than a mean. A new version that solves the same tasks in noticeably more turns is a cost regression and usually a quality warning.
• Tool error rate, split by tool. Malformed arguments to one specific tool is the classic version-change signature.
• Escalation rate to a human, per class. Moving in either direction is information: fewer escalations on the value-action class is not obviously good news.
• Abandonment, meaning runs that hit the step limit or the deadline without a terminal answer.

🔍 The diagnostic that catches the most is comparing step-count distributions between versions on the same class. Final-answer quality is expensive to judge and often looks similar; the path shows the change first.

When a plain percentage is still fine

If your agent is single-turn, read-only, and every request carries roughly the same consequence, class-based rollout is ceremony you do not need. Take the percentage and go. The ladder earns its cost the moment some fraction of your tasks can spend money, change account state, or run for twenty minutes.

The takeaway

A canary is a sampling strategy, and a sampling strategy is only as good as its match to where the risk sits. For ordinary services, risk tracks volume, so percentage works. For agents, risk tracks reversibility and chain length, both of which are anti-correlated with volume. Roll out along the axis that carries the danger, and let the easy tasks go first because they were never the question.