Rate Limits Are an Agent Design Constraint, Not an Ops Problem

The usual response to rate limiting is operational: add backoff, add a queue, request a higher quota. That works for services that make one call per request. An agent makes an unpredictable number of calls per task, so the same limit produces a failure that no amount of backoff addresses — it just moves where the run stalls.

Rate limits change what agent architectures are viable, and the constraint is best handled in the design rather than in the retry policy.

Why agents hit limits differently

Three properties conspire.

Call count per task is unbounded. A conventional endpoint makes a predictable number of downstream calls. An agent makes as many as its reasoning requires — usually five, occasionally forty. Capacity planning against an average is planning against the wrong number, because the tail is where the limit gets hit.

Concurrency is bursty by construction. A fan-out step launches ten calls simultaneously, then nothing for twenty seconds while the model thinks. Average utilization looks comfortable while the instantaneous rate blows through the limit repeatedly.

Retries stack on top. A rate-limited call gets retried by the transport, then re-attempted by the model when the error surfaces, then possibly the whole run restarts. A limit breach thus generates more traffic against the limit, which is the mechanism behind the slow spiral where a system under pressure keeps making its own pressure worse.

Design responses that work

Give the run a call budget, and let the agent see it. Not a hidden cap that errors on breach — a value in the context:

Budget: 40 tool calls per run. Used: 31. Remaining: 9.
Prioritize the remaining work accordingly.

Models behave sensibly with this information: they stop exhaustively verifying, they consolidate queries, they report partial results rather than being cut off mid-task. → A run that ends with "I checked the three most likely causes; two remain unchecked" is far more useful than one truncated at call 41.

Batch at the tool layer, not in the model's judgment. If the model wants five lookups, a lookup_many(ids: string[]) tool turns five calls into one. Batch tools are the highest-leverage change against rate limits, and they improve accuracy too — the model reasons over one coherent result set instead of five fragments.

Serialize the fan-out with a real concurrency cap. A semaphore around parallel dispatch, sized to the limit rather than to the number of subtasks. Ten parallel workers against a limit that allows three is a design error, not an ops incident.

Cache within the run. Agents re-request the same thing — the same record, the same page — several times in a long run, usually after a summarization step drops the earlier result. A per-run cache keyed on tool name plus arguments eliminates a surprising share of traffic. ⚠️ Scope it to the run: cross-run caching of mutable data introduces staleness bugs that are much harder to find than a rate limit.

When you're the one being limited

If your agent calls a partner API with a hard quota, the constraint belongs in the plan, not the retry loop.

• Reserve headroom for the tail. Size normal operation to a fraction of the quota so the occasional forty-call run doesn't exhaust it.
• Prioritize explicitly under pressure. When the budget is nearly spent, some calls matter more. Let the loop pass that ranking in rather than serving whichever call happens to be next.
• Degrade to a cheaper path. A cached or stale answer with a clear label beats a failed run, for most tasks. Making the degraded mode explicit — and telling the model it's in one — produces much better behavior than silent failure.

🔍 The metrics that make this visible

Standard API dashboards mislead here. Track instead:

• Calls per completed task, at p95 rather than mean. This is the number that predicts limit breaches.
• Peak instantaneous concurrency, not average. The burst is the problem.
• Duplicate call rate within a run — same tool, same arguments. Anything non-trivial is free savings sitting in a cache you haven't written.
• Budget-exhausted run rate. If runs regularly hit the ceiling, the ceiling is too low or the agent is inefficient; either way you now know which.

The takeaway

Backoff handles a transient limit. It doesn't handle an architecture that emits unpredictable bursts and multiplies its own retries. Give runs an explicit budget they can see and reason about, batch at the tool layer, cap concurrency deliberately, and cache within the run. Then rate limits become a parameter your design accounts for rather than an incident that recurs.