Migrating Model Versions Without Breaking Your Agent

A model upgrade is not a dependency bump. The new version is better on nearly everything and different on everything, and an agent's behavior depends on the differences in ways your tests may not cover. Teams that treat it as a version string change discover this in production.

The migration is manageable. It needs to be run as an experiment rather than a deploy.

What actually shifts

Tool selection. The most commonly disrupted behavior. A new model weighs your tool descriptions differently, so an agent that reliably chose tool A for a class of request may now choose B. Both defensible; only one matches your tests and your users' expectations.

Verbosity and step count. A model more inclined to be thorough takes more turns and produces longer outputs. Cost per task can rise substantially with no change in quality — a real and frequently unnoticed regression.

Instruction weighting. The relative pull of system prompt versus recent context shifts. Prompts tuned against one version — especially ones patched repeatedly to fix specific failures — carry accumulated compensations that may now be counterproductive.

Format adherence. Usually improves. But a downstream parser tolerant of one version's quirks can break when the quirk disappears, which is a strange failure to debug.

Refusal boundaries. Where the model declines shifts in both directions. If your agent operates near any sensitive area, this needs explicit checking.

Run it as a shadow, not a switch

The sequence that works:

1. Pin the old version explicitly. If you're on a floating alias, you're already being migrated silently and unpredictably — fix that first, regardless of the upgrade.

2. Replay your frozen eval set on both. Same inputs, same recorded tool responses, both models. This isolates model differences from everything else, and it's the only comparison that means anything.

3. Compare the right things. Not just pass rates:

                        old      new
pass rate               0.91     0.93
turns per task (p50)    4        6        ← 50% more calls
cost per task (p50)     $0.031   $0.052   ← the real headline
tool selection diffs    —        14/120 cases chose differently
format violations       3        0
refusals                1        4        ← check every one

⚠️ A version that scores higher and costs 70% more per task is a business decision, not an obvious upgrade. Most migration reports omit the cost row.

4. Read the diverging cases individually. Fourteen cases where tool selection changed is a small enough number to read. Some will be improvements. Some will reveal that your tool descriptions were ambiguous all along and the old model happened to guess your way.

5. Shadow in production. Run the new version alongside the old on real traffic without using its output. Compare trajectories and costs on live inputs. This catches the input diversity your eval set doesn't have.

6. Roll out by percentage, with the old version one config change away.

Prompts need re-tuning, not porting

The instinct is to keep prompts identical so the comparison is clean. Right for the comparison; wrong for the outcome.

Prompts accumulate compensations for a specific model's tendencies — an emphatic instruction added because the old version kept forgetting something, a formatting reminder for a quirk that no longer exists. Carried forward, these can push the new model in unhelpful directions.

✅ After the clean comparison, do a second pass: remove the workarounds and see whether behavior holds. Agent prompts frequently get shorter across a model upgrade, and a shorter prompt usually behaves more predictably.

The tests that matter most here

Safety invariants deserve special attention, because they're the ones where a small behavioral shift is unacceptable rather than merely inconvenient:

• Forbidden tools are still never called.
• Approval gates still trigger on the same conditions.
• Injection resistance is unchanged — run the injection suite against the new version specifically.
• Output schemas still validate.

These should be 100% on both versions. Anything less blocks the migration regardless of how the aggregate scores look.

🔍 After the switch

Watch for a week, on the numbers that move first:

• Turns per task — the leading indicator; it moves before cost and before complaints.
• Escalation rate — a change in either direction means the model's confidence calibration shifted.
• Tool call distribution — which tools get called, in what proportion. A shift here is the clearest sign that selection behavior changed on real inputs.

The takeaway

Treat a model upgrade as an experiment with a control: pin versions, replay a frozen eval set against both, and compare cost and turn count alongside pass rates. Read every case where the trajectory diverged, shadow on real traffic before switching, and re-tune prompts afterwards to remove the workarounds the old version needed. The upgrade is usually worth it — but "usually" is why you measure rather than assume.