The Retry Storm Your Dashboard Can't See: When Agent Failures Multiply Instead of Add

Every tool call in the run eventually succeeded. The run still took four minutes instead of forty seconds, and cost several times what the same task cost the day before. The dashboard is green: no errors logged, success rate at 100%, median call latency unchanged.

The failures did happen. They were absorbed by three retry layers that were each designed sensibly in isolation, and which compose by multiplication rather than addition.

Three retry layers nobody planned together

Layer 1 — the HTTP client. Your SDK retries connection resets, timeouts, and 429s with backoff, typically two or three times. This is invisible to your code on purpose; it is doing its job.

Layer 2 — the agent loop. A tool returns an error string, the model reads it and calls the tool again. No framework labels this a retry — it looks like the model simply taking another turn. It is the most expensive retry in the stack, because a model turn re-sends the entire accumulated context.

Layer 3 — the job wrapper. The queue, scheduler, or cron that launched the run re-runs the whole task on failure, usually two or three times.

Each layer is defensible alone. Stacked, a single genuinely broken endpoint can absorb three transport attempts inside three model turns inside three run attempts before anything reaches a human. Nobody chose that number. It fell out of three independent defaults.

Why the dashboard says everything is fine

Per-call metrics are recorded after layer 1 has finished retrying, so a call that failed twice and succeeded on the third try lands in your logs as one success. Layer 2's retries don't register as failures at all — they're extra model turns, indistinguishable from a task that legitimately needed more steps. Layer 3's restarts look like separate healthy runs.

The only signal that survives all three layers is cost and wall-clock time per task, which is exactly the metric most agent deployments don't track per completed task.

The part that makes it expensive

Here's the mechanism worth internalizing: a retry costs the context depth at which it happens, not the cost of the call that failed.

Walk a research agent through a twenty-step run — searches, page fetches, a synthesis at the end. Suppose the fetch tool fails transiently now and then, and that by step fourteen the accumulated context is fairly large, say tens of thousands of tokens.

A fetch failure at step two costs almost nothing to retry: the context is nearly empty, so the extra model turn is cheap. The identical failure at step fourteen costs a full re-send of everything accumulated so far. Same failure, same tool, same error — priced an order of magnitude apart depending on when it lands.

Now make the failure non-transient. The endpoint is down, not flaky.

• Layer 1 burns its attempts and surfaces an error → cheap, milliseconds.
• Layer 2 hands the error to the model, which tries again, twice more → three large context re-sends.
• Layer 3 decides the run failed and starts over → the agent redoes all thirteen successful steps, then hits the same dead endpoint, then repeats layer 2's three attempts.

The failure was one dead URL. The bill is the run's full context, several times over, several times over again. ⚠️ This is also how a small dependency outage turns into a token-spend spike that arrives hours before anyone notices the outage itself.

What to measure instead

Per-call dashboards cannot see this. Move the denominator from call to completed task:

• Attempts per completed task. Count every model turn and every tool invocation, including retried ones. A task that used to take twelve turns and now takes thirty-one is the alarm, even at 100% success.
• Tokens per completed task, at p95. Not the mean — retry storms are a tail phenomenon and the mean will hide them.
• Retry depth. Record where in the run a retried call happened. Failures clustered late are the expensive ones and deserve different handling than early ones.
• Succeeded-on-first-attempt as a distinct outcome from succeeded-eventually. Collapsing them is what makes the dashboard lie.

Fixes, in the order I'd apply them

1. Retry at exactly one layer, deliberately chosen. Transport blips belong to layer 1. Then make layer 2 fail fast and layer 3 not restart on tool errors at all. The default of "everything retries a bit" is the whole problem.
2. Budget the run, not the call. Give the loop a hard ceiling on total turns and total tokens, enforced by the loop itself. A run that blows its budget should stop and report, not keep grinding.
3. Distinguish retryable from terminal in the error string. The model can only route around a dead tool if the error says so. Service unavailable — do not retry, use a different source produces a different next turn than a bare stack trace.
4. Circuit-break per tool, per run. After a tool fails twice in one run, remove it from the tool list for the remainder of that run. The model then plans around its absence instead of knocking on the same door.
5. Idempotency keys on anything that writes. Layer 3 restarts replay side effects. If a step sends mail, files a ticket, or moves money, a restart without deduplication does it twice.

The takeaway

Retries were designed for stateless calls, where retrying is cheap and independent. An agent run is neither: cost grows with depth, and the layers stack. Green per-call metrics with a rising monthly bill is the signature. Count attempts and tokens per completed task, retry in one place, and give every run a ceiling it cannot exceed.