One Worker Pool or Two? Separating Fast Agent Runs From Slow Ones

A support agent answers most tickets in about eight seconds. A research agent on the same service takes six minutes, because it reads a dozen documents before it writes a word. Both are served by the same pool of workers. At 11pm nobody notices. At 10am, four research runs start inside the same minute, and every eight-second ticket queues behind them.

Nothing failed. Error rate is flat, the model provider is healthy, every run eventually returns a correct answer. The only symptom is that a job which takes eight seconds of work took ninety seconds to come back — and that symptom lands on the users least willing to tolerate it.

The failure is head-of-line blocking, not slowness

A worker running an agent session is occupied for the whole session. It is not like an HTTP handler that yields between I/O waits; it holds a conversation, a tool-call loop, and usually some in-memory state from step 1 that step 9 still needs. While a worker is inside a six-minute research run, its capacity is gone.

So a pool of 20 workers with four long runs in flight is a pool of 16 workers. With ten long runs in flight it is a pool of 10, and the fast queue's wait time does not degrade gracefully — it hinges. Below the threshold, waits are near zero. Above it, waits grow with the length of the slow jobs, which the fast jobs have no relationship to whatsoever.

That's the important asymmetry: mixing durations in one queue means your fastest work inherits the latency of your slowest work. The reverse is not true. A six-minute research run does not care that it waited four seconds behind a ticket reply.

🔍 First, confirm the workload is actually bimodal

Splitting a pool is real operational complexity. Do not do it on a hunch. The question is not "are some runs slow" — some runs are always slow. It is whether run duration has two humps.

Three things to look at, in this order:

1. A histogram of run duration, not an average. A mean of 40 seconds tells you nothing; it is equally consistent with "every run takes 40 seconds" and "90% take 8 seconds and 10% take 5 minutes." Only the second one benefits from a split.
2. Tool calls per run, bucketed. Duration is mostly step count multiplied by per-step cost. If the histogram of steps-per-run has two clusters, you have two workloads wearing one name.
3. Queue wait time correlated against concurrent long runs. This is the confirming signal. If fast-run wait time spikes exactly when the count of in-flight runs over two minutes rises, you have head-of-line blocking, not a capacity shortage.

If duration is unimodal and wait time tracks total volume instead, you have an ordinary scaling problem. Add workers and move on.

Why adding workers doesn't fix it

Adding capacity to a mixed pool works right up until it doesn't. You are buying headroom against the number of simultaneous slow runs, which is a tail statistic — it fluctuates far more than your request volume does. Doubling the pool moves the cliff, it doesn't remove it, and you now pay double at 3am when the pool is idle.

There's also a subtler cost: with one pool, autoscaling reads a single utilization number that averages two workloads together. Workers look busy, so it scales up; the new workers get filled by more long runs; the fast lane is no better off. The metric that would tell you what's wrong has been averaged out of existence.

Splitting: classify at admission, not by endpoint

The instinct is to route by feature — /support goes to pool A, /research goes to pool B. That breaks the day someone adds a "summarize these 40 tickets" button to the support surface.

Route on the properties that actually predict duration, decided at admission time:

def choose_lane(request):
    # Fast lane must be provably bounded, not just usually fast.
    if request.step_budget > FAST_MAX_STEPS:
        return SLOW
    if request.tools & SLOW_TOOLS:        # web crawl, bulk export, deep retrieval
        return SLOW
    if request.input_tokens > FAST_MAX_INPUT:
        return SLOW
    return FAST

# And enforce it, so a misclassification is bounded rather than infectious.
run(request, lane=lane, max_steps=LANE_LIMITS[lane].steps,
    deadline=LANE_LIMITS[lane].wall_clock)

The enforcement line matters as much as the routing. A fast lane is only fast if a run that lies about itself gets cut off at the deadline and re-admitted to the slow lane. Without that, one mis-scoped run reintroduces exactly the blocking you split the pool to prevent.

Sizing the two pools

• The fast pool is sized for arrival rate, because its service time is nearly constant. Standard queueing intuition applies and modest headroom is enough.
• The slow pool is sized for concurrency, and capped. Its ceiling is usually not your compute — it's the rate limit on the model API or the downstream system the long runs hammer. Cap it deliberately and let the excess wait in a queue where waiting is acceptable.
• Don't let them borrow from each other in the direction that hurts. Slow work spilling into fast capacity recreates the original problem. Fast work borrowing idle slow capacity is fine.

⚠️ When one pool is still the right answer

Two pools is not automatically the better design.

• Low volume. If you rarely have more than one or two runs in flight, blocking is theoretical. Keep the simpler system.
• Runs are slow because they're waiting, not working. If the six minutes is mostly the agent blocked on a slow API or a human approval, the right fix is to stop occupying a worker at all: checkpoint the run, release the worker, and resume on the callback. Durable/suspendable execution beats a second pool, because it fixes the waste instead of relocating it.
• You can't classify at admission. If nothing about a request predicts its duration better than chance, a split just distributes the problem across two queues. Invest in a duration signal first.

The takeaway

The decision is not "how many workers do we need." It's whether one queue's latency budget is allowed to depend on another queue's work. A shared pool quietly says yes. If your slowest workload is minutes long and your most latency-sensitive workload is seconds long, that answer is wrong, and no amount of capacity will make it right — because the thing you're actually short of is isolation, not compute.