Testing Multi-Agent Systems Without Losing Your Mind

Testing a single agent is manageable: freeze the tool responses, run the case, assert on the trajectory. Add a second agent and the combinatorics turn hostile — each agent's output is the next one's input, so variation compounds and a single end-to-end assertion tells you almost nothing about where things went wrong.

The way through is to stop testing the system as one unit.

Test each agent as a unit, with contracts at the seams

Every sub-agent has an implicit contract: given input of this shape, return output of that shape, satisfying these properties. Make it explicit and test it in isolation.

# Unit test for the extractor sub-agent
result = extractor.run(document=FIXTURE_INVOICE)

assert validates(result, EXTRACT_SCHEMA)
assert result.total == sum(line.amount for line in result.lines)
assert all(l.quote in FIXTURE_INVOICE for l in result.lines)   # grounded
assert result.currency in {"EUR", "USD", "GBP"}

No supervisor, no downstream consumer, no compounding variation. When this fails you know exactly which component is wrong — which is the property the end-to-end test destroys.

⚠️ The contract must be enforced at runtime too, not only in tests. A sub-agent returning something off-contract in production should fail at the boundary rather than propagating a malformed result three hops downstream.

Test the supervisor against stubs

The supervisor's job is delegation, aggregation, and handling what comes back. None of that requires real workers.

Replace every worker with a stub returning a fixed contract-valid result, and test the decisions:

• Does it delegate the right items to the right workers?
• Does it aggregate correctly when all workers succeed?
• What does it do when one returns needs_input?
• What when one fails? When one times out? When two disagree?
• Does the final output state which items failed?

These stubs are fast, deterministic, and free. → Most supervisor bugs are in the unhappy paths, and stubs are the only practical way to exercise all of them.

Then test integration on a small number of cases

End-to-end tests still earn their place — they catch contract mismatches and emergent behavior nothing else sees — but they should be few, and they should assert on properties rather than on exact outputs:

assert result.items_processed == 12
assert result.failed_items == []
assert no_cycles(result.delegation_graph)
assert result.total_agent_calls <= 30
assert provenance_complete(result)     # every finding traces to a worker

Those hold regardless of which path the system took. An assertion on the exact final text will fail on every harmless variation and get muted within a month.

The four failures only integration finds

Worth writing specific cases for, because unit tests structurally cannot catch them:

Contract drift. Worker changes its output shape; supervisor still parses the old one. Runtime contract enforcement catches this in production; an integration test catches it before.

Compounding degradation. Each hop is individually acceptable and the result three hops down is unusable. Test by asserting on provenance depth — how many transformations separate a final claim from the tool result that produced it.

Coordination failures. Cycles, orphaned waits, livelock. Assert on the delegation graph shape and on total call count.

Cost blowup. Individually reasonable agents that collectively cost ten times the budget. ✅ Assert a ceiling on total calls and total tokens in every integration test — it's one line and it catches an entire class of regression that quality assertions miss completely.

🔍 Record real runs as fixtures

The fastest way to a useful suite: capture a real multi-agent run and save each agent's inputs and outputs as fixtures.

Now each agent can be unit-tested against genuine inputs from its neighbours, and the supervisor can be tested against real worker outputs rather than idealized stubs. Real outputs are messier than anything you'd invent, and that mess is where the bugs are.

Keep both, though — recorded fixtures for realism, hand-written stubs for the failure cases that don't occur naturally in a captured run.

Keeping it affordable

Multi-agent integration tests are slow and expensive, which is why they get skipped.

• Unit tests on every commit. Fast, cheap, catch most issues.
• Supervisor stub tests on every commit. Effectively free.
• A handful of integration cases per merge, with a strict cost ceiling.
• The broad matrix nightly, not per commit.

The takeaway

Don't test a multi-agent system as one thing. Give each agent an explicit contract and unit-test it against recorded inputs; stub the workers and test the supervisor's decisions, especially the unhappy ones; keep integration tests few and assert on invariants — graph shape, call count, provenance, failed-item reporting. And put a token ceiling in every integration test, because collective cost blowup is the regression nothing else notices.