Flaky Agents: Separating Model Nondeterminism From Your Bugs

A test fails. You re-run it and it passes. With ordinary software that's a bug you haven't found yet. With agents, the same evidence is ambiguous — it might be a real defect, or it might be the model taking a different valid path. Treating both the same way is how agent test suites get abandoned.

There's a procedure for telling them apart, and it takes one afternoon to set up.

First, remove the variance you control

Before analyzing anything, eliminate the sources of nondeterminism that aren't the model:

Freeze the tool responses. A test hitting live services varies because the world varies. Record real responses once and replay them. This is the single largest source of apparent flakiness and the easiest to remove.

Freeze the clock and any IDs. "Yesterday" changes meaning overnight; generated IDs differ every run. Inject both.

Pin the model version. A provider's -latest alias moves under you, and behavior shifts with it. Pin explicitly, and treat version bumps as a change to be evaluated, not absorbed.

Set temperature deliberately. Zero doesn't guarantee determinism — batching and floating-point non-associativity still produce variation — but it removes the deliberate kind.

After these four, remaining variation is genuinely the model. ⚠️ Skipping this step means every subsequent analysis is measuring your infrastructure, not your agent.

Second, measure the variation instead of arguing about it

Run each case ten times and record the distribution rather than a pass/fail.

case: refund_within_window
  passes:      9/10
  tool_seqs:   {[lookup, check, refund]: 7,
                [check, lookup, refund]: 2,
                [lookup, refund]:        1}   # <- the failure
  turns:       p50=4  p95=6

That output distinguishes the two situations immediately. Seven-versus-two on ordering is harmless path variation. The one run that skipped check is a real defect — it produces a different outcome, not a different route to the same outcome.

The rule: variation in path is expected; variation in outcome is a bug. Assertions should target outcomes and safety invariants, so this distinction shows up automatically rather than requiring interpretation.

Third, classify what's left

For each case where outcomes genuinely differ:

• Fails rarely, on one specific input → a real edge case. The model is finding an ambiguity in your prompt or tool descriptions that most inputs don't expose. Highest-value bug in the suite.
• Fails at a consistent rate across many inputs → an under-constrained task. The request or the tool surface admits multiple readings. Fix by constraining, not by retrying.
• Fails only under load or with long context → not model variance at all. Look for truncation, timeouts, or a summarization step firing at a threshold.
• Fails since a specific date → provider-side change. This is why version pinning matters, and why the flip date is worth recording.

The reporting change that keeps the suite alive

Stop reporting agent tests as binary. Report rates with thresholds:

✅ never_calls_delete:        10/10 required, got 10/10
✅ refund_requires_check:     10/10 required, got 10/10
⚠️  completes_within_6_turns:  8/10 required, got 7/10   REGRESSED (was 9/10)

Safety invariants demand 10/10. Efficiency and style targets get realistic thresholds. And the most useful column is the comparison against the previous run — a case that drops from 9/10 to 7/10 hasn't failed yet and is the earliest signal you'll get that something changed.

🔍 Track flip rate over time as a first-class metric. Rising variance on previously stable cases usually means a prompt has grown ambiguous or a tool description has drifted, and it precedes visible failures by a comfortable margin.

What not to do

• ❌ Retry until green. It converts a signal into a delay and guarantees the defect ships.
• ❌ Assert on exact tool sequences. Every harmless reordering becomes a failure and the suite gets muted.
• ❌ Run each case once and treat the result as truth. A single sample from a stochastic process is not a measurement.
• ❌ Raise temperature to "explore more" in tests. You want the production configuration, measured, not a different one.

The takeaway

Agent flakiness is two things wearing one costume: infrastructure variance you can eliminate, and model variance you must measure. Freeze the clock, the tools, and the model version. Then run every case ten times and assert on outcomes rather than paths. Once the suite reports rates instead of booleans, "it failed once" becomes a number you can act on rather than an argument you keep having.