A 1% Flaky Tool Shouldn't Take You Down. Three Nested Retry Layers Will.
A tool that fails one call in a hundred is healthy. The retry logic wrapped around it is what turns a dependency's bad minute into a queue of hung user requests — and the load multiplier is at its worst exactly when the dependency is weakest.
A tool that fails one call in a hundred is a healthy tool. Wrap an agent around it, give every layer permission to retry, and that same tool can produce twenty minutes where every user request hangs and half of them come back wrong. The underlying failure rate never moved. The amplification did.
Retries are the most common reliability control in an agent stack and the least examined one. They get added at three different altitudes by three different people, each of whom believes they are the only one retrying. Here is what that compounding actually does.
🔍 The first multiplication: your SLO is per call, your user lives in a run
A single tool call failing 1% of the time sounds like a rounding error. But nobody experiences a call. They experience a run — a support agent that pulls the account, checks entitlement, fetches three recent tickets, queries billing, and drafts a reply.
Say that run makes twelve tool calls against the same flaky dependency. If failures were independent, the chance a run hits at least one is 1 - 0.99^12, or about 11%. One in nine conversations touches the failure. That is the honest baseline before anyone writes a single line of retry code, and it is the reason retries feel obviously correct.
They are correct. What is not correct is having three of them.
The second multiplication: nobody owns the retry
A typical stack, top to bottom:
| Layer | Who added it | Attempts |
|---|---|---|
| HTTP client / vendor SDK | The SDK author, by default | 3 |
| Tool wrapper in your codebase | The engineer who hit a timeout once | 3 |
| Agent loop error handling | The framework, or your own on_tool_error |
3 |
Nobody chose 27 attempts. 27 attempts is what the stack does.
The expected number of real network calls per logical request, with failure probability p and three attempts per layer, is roughly (1 + p + p²)³:
p = 0.01 -> 1.03x load
p = 0.30 -> 2.69x load
p = 0.70 -> 10.5x load
Read that bottom row again. The load multiplier is smallest when the dependency is healthy and largest when it is already dying. Your retry logic is a load amplifier that switches on precisely when the thing it is protecting has no capacity left. A dependency that could have recovered from 30% errors in ten seconds instead gets buried, and stays buried after the original trigger is gone. That self-sustaining state is the whole reason a 1% tool becomes an outage rather than a blip.
The third multiplication: an agent's retry is not a retry
Client-library retries are boring and predictable — same request, same arguments, backoff, done. The agent loop's "retry" is something else entirely. When a tool error lands in the model's context, the model re-reasons. It may:
- Call the same tool with different arguments, because it assumes the arguments caused the error.
- Call a different tool that looks adjacent, producing a plausible answer from the wrong source.
- Decide the data is unavailable and answer from what is already in context, which may be three turns stale.
- Announce success having silently dropped a step.
None of these register as failures anywhere. Your tool error rate looks fine. Your run completion rate looks fine. The thing that degraded is answer correctness, and the only place it surfaces is a user saying the agent told them something untrue.
⚠️ This is the part teams miss when they instrument retries: the dangerous outcome of an agent-level retry is not another error, it is a confident wrong answer.
Latency is the outage your users actually see
Nested backoff compounds worse than nested attempts, because each layer waits.
An inner layer with 1s, 2s, 4s backoff burns up to 7 seconds before it reports failure upward. The middle layer treats that 7 seconds as one attempt and runs three of them with its own waits — call it 25 seconds. The outer agent loop does the same to the middle layer, and now a single flaky call has consumed roughly 80 seconds.
A run budgeted at 30 seconds does not fail at 80 seconds. It sits there. Connections stay open, the queue backs up, and requests that never touched the flaky tool start timing out behind the ones that did. To the user this is not "one integration is degraded." It is "the product is down."
The quiet one: retrying a write
Everything above assumes the call is safe to repeat. Layer 2 retrying a create_refund that actually succeeded but timed out on the response is not an availability problem, it is a money problem. Any tool that mutates state needs an idempotency key threaded through every layer that might repeat it — and if you cannot guarantee that, the tool must be marked non-retryable and fail loudly on first error.
✅ The fix: one owner, a budget, and honesty upward
- Pick exactly one retry layer and disable the others. Usually the tool wrapper, because it is the only layer that knows both the transport and the semantics. Set the SDK's retry count to zero explicitly. Configure the agent loop to surface tool errors rather than re-attempt them.
- Budget, don't count. Retries should be governed by a deadline shared across the whole run, not by a per-call attempt number. Before each attempt: if
remaining_deadline < expected_latency * 2, do not retry — give up now and leave time to degrade gracefully. - Classify before retrying. Timeouts, 429s and 503s are retryable. A 400, a schema violation, or "record not found" is not — retrying those is pure waste that also teaches the model the tool is unreliable.
- Add a circuit breaker per tool, and tell the agent it tripped. Not a raw error. A structured message: this tool is unavailable, it will not be retried this run, proceed without it or stop. An agent that knows a capability is gone can replan. An agent that gets a generic error will keep poking.
- Instrument attempts, not calls. Emit total attempts per logical request and per-tool retry rate. A ratio drifting above 1.05 is a dependency going soft, visible long before the error rate moves.
- Test at the failure rate that matters. Injecting 1% faults proves nothing. Run your eval suite against a tool stub at 30% and 70% failure and watch what the run does — that is where the amplification lives.
The point is not that retries are wrong. It is that in an agent stack the retry is the highest-leverage piece of load-bearing code nobody reviewed, and it was written three times by three people who never talked. Find the layers, delete two of them, and give the survivor a deadline.