Queueing and Backpressure for Agent Workloads

Agent workloads break the assumptions most queue configurations are built on. A conventional job has a roughly predictable duration and cost; an agent run can take four seconds or four minutes, consume one model call or forty, and the difference isn't visible until it's finished.

Sizing a worker pool against an average, when the distribution has a long tail, produces a system that looks fine until it doesn't.

Why the usual settings misbehave

Duration variance is enormous. With a p50 of ten seconds and a p99 of four minutes, a pool sized for the median stalls whenever a few tail runs arrive together — and they do arrive together, because hard inputs correlate.

The real constraint is upstream. Your bottleneck is usually the model provider's rate limit, not your CPU. Adding workers past that point increases queue depth and error rates while doing nothing for throughput. This is the most common misconfiguration in agent deployments.

Cost, not just capacity, is at stake. An overloaded conventional service gets slow. An overloaded agent service can get slow and expensive, because timeouts trigger retries that re-send full contexts.

Work isn't uniform. Two items in the same queue can differ by two orders of magnitude in cost. FIFO treats them identically.

Separate queues by shape

The single highest-value change: stop putting fast interactive work and slow batch work in one queue.

interactive:  user waiting, short, high priority, small pool, strict timeout
batch:        no one waiting, long, low priority, larger pool, generous timeout
retry:        previously failed, lowest priority, capped attempts

Separate pools mean a batch of two hundred documents can't starve the person waiting for an answer. ⚠️ Sharing one queue is how a nightly job makes the product feel broken every night, and the correlation takes ages to spot.

Size the pool to the upstream limit

Work backwards from the provider's rate limit rather than from your machines:

provider limit:        400 requests/min
avg model calls/run:   8
                       → ~50 runs/min sustainable
avg run duration:      25s
                       → 50 × 25/60 ≈ 21 concurrent runs

Then set the pool a little below that and let the queue absorb bursts. Workers beyond this number are queue depth wearing a costume. Recompute when your average call count changes — a prompt edit that adds two turns per run cuts your sustainable throughput by a fifth.

Backpressure that means something

When the queue grows, doing nothing means every user waits and nobody is told. Three responses, in preference order:

Reject early with a clear signal. For interactive work, a fast "system is busy, try shortly" beats a ninety-second wait ending in a timeout.

Shed by priority. Drop or defer the lowest-value work first — background enrichment, speculative pre-computation, anything nobody is waiting for.

Degrade the work itself. Under load, run a reduced version: fewer turns allowed, a smaller model on the cheap steps, cached answers where acceptable. ✅ Tell the user they got the degraded version. A quietly worse answer is a trust problem; a labelled quick answer is a feature.

Timeouts at every layer, and they must agree

Agent runs need timeouts at four levels, and misconfigured relationships between them cause the strangest incidents:

• Per model call.
• Per tool call.
• Per run — total wall clock.
• Per queue item, after which it's considered abandoned.

The rule: the queue's visibility timeout must exceed the maximum run duration. Otherwise a slow-but-healthy run gets redelivered to a second worker while the first is still going, and now the same task executes twice — with duplicated side effects. This is a genuinely common bug and it presents as mysterious duplicates rather than as a timeout problem.

🔍 What to watch

• Queue depth by queue, not overall.
• Age of the oldest item — more actionable than depth, because it directly reflects user experience.
• Worker utilization, to tell "not enough workers" from "upstream limited."
• Rate-limit rejections from the provider. Any sustained level means the pool is oversized, and adding workers is actively harmful.
• Duplicate execution count, which should be zero and tells you your timeout relationships are wrong when it isn't.

The takeaway

Size the pool to the model provider's limit rather than to your hardware, split queues by work shape so batch can't starve interactive, and make backpressure explicit — reject fast, shed by priority, or degrade and say so. Then check that your queue visibility timeout is longer than your longest run, because the alternative is the same expensive task running twice and nobody knowing why.