#latency
Every post tagged "latency".
25 posts · showing 1–10
-
A 1% Flaky Tool Shouldn't Take You Down. Three Nested Retry Layers Will.
A tool that fails one call in a hundred is healthy. The retry logic wrapped around it is what turns a dependency's bad minute into a queue of hung user requests — and the load multiplier is at its worst exactly when the dependency is weakest.
-
One 300-Second Timeout Tells You Nothing. Agents Need a Deadline Budget.
A run that dies at its ceiling with no attribution tells you nothing about which part was slow — and pushes you toward the laziest fix, raising the ceiling. Three nested clocks, an inherited remainder, and a reserved tail slice turn timeouts into a signal you can actually read.
-
Cached or Re-Called? Giving Every Agent Tool a Staleness Budget
An agent that reads a value once and acts on it four minutes later isn't reasoning badly — nobody told it how long that value stays true. Per-tool staleness budgets, and the one rule that matters most before a write.
-
One Worker Pool or Two? Separating Fast Agent Runs From Slow Ones
A handful of long research runs can starve hundreds of eight-second ticket replies without a single error firing. The fix isn't more workers — it's deciding, at admission, which lane a run belongs in.
-
Draining a Data Backlog: Should the Agent Batch Rows or Take Them One at a Time?
The ratio of preamble tokens to row tokens decides whether chunking a backlog saves an order of magnitude or nothing at all — and it's the alignment, retry amplification and blast radius that decide whether you should anyway.
-
Retry, Reroute, or Hand Back: Budgeting an Agent's Second Chances
A tool call fails and the loop tries again — five times, forty seconds, and still nothing usable for the user. Retrying is only one of three legal responses, and picking the wrong one turns a fast failure into an expensive one.
-
IDE Assistants vs Agent Harnesses: Different Products, Different Futures
They get discussed as one category and have opposite requirements on latency, context, failure cost, and model tier. One is commoditizing into a feature; the other's differentiators aren't AI capabilities at all.
-
What Runs Locally Today and What Still Won't Soon
Local handles the steps; hosted handles the orchestration. The agent loop is the hardest thing to localize, because it stresses all three local constraints at once.
-
When a Small Model Beats a Big One, and How to Tell
Sometimes "beats" means cheaper at acceptable quality — and sometimes it means genuinely better on the metric that matters. Four cases where smaller wins outright, and four where it looks like it should and doesn't.
-
Matching Model Tier to Task: A Practical Table
"Simpler tasks get cheaper models" isn't actionable, because simplicity isn't the predictor. Two properties are — and one row of the table overrides all the others.