Your Agent Fleet Kept Its Own Quota Pinned for Forty Minutes: Synchronised Retries Are a Self-Inflicted Outage

The first 429 was the provider's fault. Every 429 after that was yours.

Here is the shape of it. An ops agent fleet — a few dozen workers triaging alerts, each one running a short tool-using loop — brushes against the model provider's rate limit during a busy hour. The client library does the sensible thing and retries. Forty minutes later the fleet is still failing, the alert backlog is deep, and the provider dashboard shows request volume higher than when the trouble started. Nobody deployed anything. The load never spiked. The fleet simply found a rhythm and could not get out of it.

⚠️ The mechanism: a rate limit is a clock

A rate limiter is a shared clock. When it rejects a batch of requests, it does not reject them at random times — it rejects them all at the same instant, the instant the window filled. Every worker that was mid-flight gets its 429 within milliseconds of every other worker.

Now each of those workers starts a retry timer. If the timer is deterministic — fixed delay, or plain exponential backoff with no randomness — then every worker's second attempt lands in the same millisecond too. The limiter rejects them all again, and now they are more tightly synchronised than before, because the failure itself acted as a barrier that lined them up.

Plain exponential backoff makes this worse, not better, in one specific way: it thins the traffic but it does not spread it. Thirty-two workers retrying at 1s, 2s, 4s, 8s produce four narrow spikes of thirty-two requests instead of a smooth trickle. The limiter sees a spike and rejects the spike. Between the spikes, your quota sits unused.

Agents amplify this beyond what ordinary API clients do, for three reasons:

• A run is many calls, not one. A single triage task might be eight model calls. One 429 on call six does not fail one request — it either retries at depth or throws away the whole run's work, and the run comes back to redo all eight.
• Retries nest. The SDK retries the HTTP call. The agent loop retries the step. The job queue redelivers the task. Three independent layers each retrying three times is twenty-seven attempts from one logical unit of work, and nobody wrote "27" anywhere.
• The work is elastic. A queue-fed fleet has a backlog. The moment capacity returns, every waiting task starts at once — and re-synchronises the whole fleet on the recovery edge.

🔍 How to tell this is what you are looking at

Three signals, in order of how quickly you can check them:

1. Request rate rises while success rate falls. Load shedding that works looks like fewer requests. A retry storm looks like more requests and fewer completions.
2. Requests arrive in a comb, not a curve. Bucket your model-call timestamps at 100ms. Healthy traffic fills most buckets. A synchronised fleet leaves most buckets empty and a few enormous.
3. Your 429 count exceeds your task count. If forty tasks produced four hundred rejected calls, the retries are the load.

If you only look at the provider's aggregate graph, all three are invisible. The comb is the giveaway, and it only shows up at sub-second resolution.

The fix has three parts, and skipping any one leaves the loop intact

1. Jitter, and make it full jitter. Not delay = base * 2^n. Not delay = base * 2^n + random(0, 100ms), which is a spike with fuzzy edges. Use the whole interval:

# full jitter — the delay is uniformly random across the window
delay = random.uniform(0, min(cap, base * 2 ** attempt))

Sampling across the entire backoff window is what actually decorrelates workers. A small additive jitter on a large delay barely moves anything.

2. Honour Retry-After, and treat it as a floor rather than a suggestion. When the provider tells you when to come back, retrying earlier is guaranteed to fail and still counts against you on most limiters. Take the header value, then add jitter on top — otherwise every worker that got the same header comes back at the same moment, which is exactly the problem you started with.

3. Put one limiter in front of the model client, not one per worker. This is the part teams skip, and it is the part that matters. Per-worker backoff controls how often one worker asks; it cannot control how many workers ask together. A shared token bucket — in a small sidecar service, in Redis, in whatever the fleet already shares — converts "thirty-two workers guessing" into "one component that knows the budget."

before →  worker ─┐
          worker ─┼─→ provider   (32 independent guesses about the limit)
          worker ─┘

after  →  worker ─┐
          worker ─┼─→ shared token bucket ─→ provider   (one budget, enforced once)
          worker ─┘

With a shared bucket, a 429 stops being a signal each worker reacts to independently and becomes a signal you handle once, centrally, by lowering the fill rate.

Two things to add while you are in there

Budget the retries per task, not per call. Give a run a total attempt allowance — say twelve model calls including retries — and fail it loudly when it is spent. Otherwise nested retry layers multiply silently and you discover the multiplication during an incident.

Stagger the recovery. When capacity comes back, do not release the backlog at once. Ramp the token bucket's fill rate back up over a minute or two. A fleet that recovers instantly re-synchronises instantly, and you get the same outage a second time with a cleaner conscience.

The takeaway

The uncomfortable part of a retry storm is that every individual component behaved correctly. Each worker retried a failed call a reasonable number of times with a reasonable delay. The failure lives entirely in the correlation between them — which is invisible in any single worker's logs and invisible in any code review of a single worker's retry policy.

That is the general lesson for agent fleets: the interesting failures are not in what one agent does, but in what all of them do at the same instant. Design the retry policy at the fleet level, or the fleet will design one for you.