Latency Budgets for Agents: Where the Seconds Actually Go

"The agent is slow" is a symptom with at least five distinct causes, and the usual response — try a faster model — addresses one of them. Building a latency budget takes an hour and tells you which one you have.

Decompose a run

For a run taking twelve seconds, split the time:

time to first model response       1.4s   (queue + prefill + first token)
model generation (4 calls)         3.2s
tool execution (6 calls)           5.8s   ← largest single share
context assembly (retrieval, etc)  1.1s
overhead (serialization, logging)  0.5s

Just producing this table redirects most optimization efforts. Here the model is under 40% of the time, and swapping in a faster one addresses at most that fraction.

The five causes and their fixes

1. Too many turns. The dominant factor in most slow agents, because each turn is a full round trip. A run taking eight turns for work that needs three is slow for reasons no model change fixes.

Fixes: batch tools so one call replaces five, parallel tool execution within a turn, better tool descriptions so the agent doesn't take exploratory steps, and a clear termination signal so it doesn't verify work it already did.

2. Slow tools. Frequently the largest share and the least examined, because tool latency lives in someone else's system.

Fixes: caching within the run, parallel execution, and — the big one — designing tools that answer the actual question. A single get_order_summary returning what's needed beats four calls the agent assembles itself.

3. Prefill on a large context. Time-to-first-token scales with input size. A forty-thousand-token context costs meaningfully more before generation even starts, on every turn.

Fixes: prompt caching, which is the single highest-leverage latency optimization available and often already broken by an unstable prefix; trimming what's sent; and moving large results out of context and into references.

4. Generation length. Long outputs take proportionally long. An agent that narrates its reasoning verbosely on every turn pays for it in wall clock.

Fixes: ask for structured output rather than prose where the output feeds another step, and don't request explanations nobody reads.

5. Queueing. Under load, time waiting for a worker or a rate limit. Invisible in per-call metrics and obvious in end-to-end.

Fixes: capacity, prioritization, and separating interactive from batch work.

Budget by user experience, not by component

Work backwards from what a person will tolerate:

interactive, user waiting:      < 3s to first visible output
                                < 15s to completion
                                → 3-4 turns maximum, aggressive caching

user waiting but expecting work: < 60s
                                → 8-10 turns, show progress

background:                      minutes acceptable
                                → optimize cost, not latency

⚠️ These are different products. Trying to make one agent serve all three usually produces something that satisfies none. Route by expectation and design each path to its own budget.

💡 The perception levers

Some of the best returns aren't reductions in actual latency:

Stream the first thing you know. Progress events from the loop — which step, which tool — fill the silence during tool execution, which is where the wait actually is.

Return partial results. An agent checking six services can report the first three while it works. Substantially better than nothing for twelve seconds.

Do the fast path first. If a deterministic rule answers most requests, answering those in milliseconds changes the perceived latency of the whole system, even though the hard cases still take twelve seconds.

🔍 What to measure

• End-to-end p50 and p95 per feature, not overall.
• Time-to-first-output, separately from completion.
• The breakdown above, per run, so slow runs can be classified without investigation.
• Turns per task — the strongest single predictor.
• Cache hit rate, since a drop shows up as latency before anyone connects the two.

The p95 matters more than for ordinary services, because agent latency distributions have long tails and the tail is where people form their impression.

The takeaway

Measure the split before optimizing. Tool execution and turn count usually dominate; the model is a smaller share than the instinct to swap it suggests. Reduce turns with batch tools and parallel calls, cache the prefix, move large results out of context, and set separate budgets for interactive, waiting, and background paths. Then stream progress and return partials — half the perceived problem is silence, not seconds.