Upgrading One Agent in a System of Six
A contained improvement to one classifier made the supervisor above it drop half its escalations. No error, no failing test — because each agent's own tests passed against its own version.
A classifier agent in a support pipeline was improved and deployed on its own, as a contained change. Within a day the supervisor above it was dropping roughly half the escalation flags it should have acted on.
The improved classifier had started returning severity as an object with a score and a rationale, where it used to return a bare string. The supervisor read severity as a string, got something it didn't recognize, and treated the case as unclassified. Nothing errored. No test failed, because each agent's own tests passed against its own version.
Multi-agent systems are distributed systems with softer contracts
Between two services, the contract is a schema, and a breaking change is a caught error at deserialization. Between two agents, the contract is frequently a prose description plus a loosely validated JSON blob — and the consumer is often a model, which is exceptionally good at proceeding sensibly when given something slightly wrong.
That tolerance is the problem. A model handed an unexpected shape doesn't crash; it does something reasonable-looking with what it recognizes and ignores the rest. → The degradation is silent by construction, and it scales with how many agent-to-agent boundaries you maintain.
Three kinds of skew, in increasing order of nastiness
Shape drift. A field is added, renamed, or nested. The consumer ignores what it doesn't recognize. Visible if you look; invisible if you don't.
Semantic drift. The field name and type are unchanged, and the meaning moved. A confidence score recalibrated so 0.7 now means what 0.9 used to. A severity scale that gained a level. Nothing in any schema check can catch this, and every downstream threshold is now wrong by an amount nobody has measured.
Behavioral drift. The worker's distribution changed. It now escalates cases it used to resolve, so the supervisor sees twice the volume in a queue sized for the old rate. The contract holds perfectly; the system still degrades.
Version the contract, not just the agent
Deploying agents independently requires the same thing deploying services independently requires: an explicit, checkable contract with a version on it.
{ contract_version: "2",
case_id: "...",
severity: {score: 0.82, level: "high", rationale: "..."},
... }
The consumer declares what it accepts, and a mismatch fails loudly at the boundary:
ACCEPTS = {"1", "2"}
if result.contract_version not in ACCEPTS:
raise ContractMismatch(producer=result.contract_version, accepts=ACCEPTS)
⚠️ Tolerating unknown extra fields is correct and useful. Tolerating a missing or reshaped required field is what turns a deploy into a silent outage. Those two behaviors get conflated under "be liberal in what you accept," and only the first one should apply.
Roll out consumers first
The ordering that works, in every case:
- Update the consumer to accept both the old and new shapes. Deploy. Nothing changes yet.
- Update the producer to emit the new shape. Deploy.
- Once no producer emits the old shape, remove the old branch from the consumer.
Reversed, the system breaks the moment the first upgraded worker returns a result. This is unremarkable API practice, and it gets skipped for agents because "improving the classifier" doesn't feel like an interface change. It is one.
Semantic drift needs a rename, not a version bump
The trap worth naming separately: if a field keeps its name and type but changes meaning, no amount of version negotiation helps a consumer that reads it the old way.
✅ Recalibrating a confidence score means introducing confidence_v2, or bumping the contract in a way the consumer must explicitly acknowledge before the new values reach it. Changing what the number means in place, under the same name, is the change most likely to go undetected for months — and every threshold tuned against the old scale is quietly wrong from the moment it ships.
🔍 Detecting skew before someone notices
Log conformance at every boundary:
- The producer's contract version and the consumer's accepted set, on every handoff.
- Fields the consumer ignored. This is the single best early-warning signal in a multi-agent system. A field appearing in results that nobody downstream reads means either a producer got ahead of a consumer, or you're paying to generate something unused.
- Distribution of key output values over time. Severity mix, escalation rate, confidence histogram. A shift with no deploy on the consumer side is behavioral or semantic drift, and it's the only way to see either.
Test the matrix, not just the agents
Each agent passing its own tests is exactly what let this through. Add a small cross-version matrix to CI:
supervisor v1 supervisor v2
classifier v1 ✓ ✓
classifier v2 ✗ expected ✓
The expected failure matters as much as the passes — it proves the mismatch is detected rather than silently absorbed. Two agents and two versions is four cases, which is cheap. That the number grows with topology size is another argument for keeping the agent count small.
The takeaway
An agent is an interface, and improving it is an interface change. Put an explicit version on every inter-agent contract, fail loudly on unrecognized versions while tolerating unknown extra fields, roll consumers out before producers, and never recalibrate a field's meaning under its existing name. Then log what each consumer ignored — that number moves before anything visible breaks.