Canary Releases for Prompt Changes

A prompt edit is a behavior change to your most complex component, shipped as a string. It gets code-reviewed by people reading English rather than logic, tested against a suite that covers a fraction of real inputs, and deployed to everyone at once.

Canarying prompt changes is straightforward, and the reason to do it is that prompt edits have unusually wide and unpredictable blast radius: a sentence added to fix one behavior routinely changes three others.

Version the prompt as an artifact

Nothing works until prompts have identities:

prompt_id:      support_agent
version:        v14
hash:           sha256:9f2c...
parent:         v13
changed:        "Added instruction to verify account tier before refunds"

The hash matters because prompts are frequently assembled from templates, and the rendered string is what the model saw. Logging the template version without the rendered hash means you can't reproduce what actually happened.

Every model call records the prompt version. Without that, a canary tells you two populations differ but not which change caused it.

Split by a stable key

Route a percentage of traffic to the new version, keyed on something stable — user ID or tenant, not run ID.

Keying on the run means a user gets v13 on Monday and v14 on Tuesday, and any feedback they give is uninterpretable. Stable assignment also means a user's experience is consistent, which matters if the change alters tone or format.

Start small — a few percent — and hold long enough to accumulate enough runs for the comparison to mean anything. ⚠️ For a low-traffic agent, that can be days. Ending a canary early because it "looks fine" after twenty runs is the most common way this goes wrong.

What to compare

Not just quality. Prompt changes move several things at once:

                          v13      v14
completion rate           0.94     0.93
turns per task (p50)      4        5      ← more work per task
cost per task (p50)       $0.028   $0.041 ← 46% more
escalation rate           0.06     0.11   ← escalating more
tool distribution         —        refund_order   -22%
                                   check_tier     +140%   (intended)
user rephrase rate        0.09     0.08

The tool distribution row is the most informative and the least often collected. A prompt change that shifts which tools get called is doing something structural, and comparing distributions tells you whether it's what you intended. Here check_tier rising was the goal; refund_order falling by a fifth was not necessarily, and needs a look.

The user rephrase rate is the closest thing to a free quality signal — users re-asking is a labelled failure that costs nothing to collect.

Guard the invariants absolutely

Some things don't get a percentage comparison. Before any traffic reaches a new prompt version, the eval suite must show:

• Forbidden tools still never called.
• Approval gates still trigger.
• Output schema still validates.
• Injection suite still passes.

These are pass/fail gates, not metrics to compare. ✅ A prompt change that improves completion rate and weakens an injection defense is not a trade to evaluate; it's a blocked release.

Automate the rollback trigger

Define the abort conditions before starting, as numbers:

rollback_if:
  completion_rate      < baseline - 0.03
  cost_per_task        > baseline * 1.25
  escalation_rate      > baseline * 2
  any_invariant_fails  == true

Written in advance, these get honoured. Decided during the canary, they get rationalized — the cost increase becomes acceptable because the change was someone's idea and the quality looks slightly better.

🔍 Read the diverging runs

Statistics tell you that something changed. To know what, take a sample of cases run under both versions and read the pairs where the trajectory differed.

This is where you learn that the new instruction is being applied in cases you didn't intend, or that a sentence meant to encourage verification is being read as a requirement to verify everything. Ten read pairs teach more than any aggregate.

The takeaway

Prompts are behavior, so ship them like behavior: versioned with a hash of the rendered string, assigned by a stable key, compared on cost and turn count and tool distribution as well as quality, gated absolutely on safety invariants, and rolled back on thresholds written before the canary started. Then read the runs that diverged — that's where the actual effect of your edit is visible.