#latency
Every post tagged "latency".
25 posts · showing 11–20
-
Should You Always Reach for the Flagship Model?
"Always use the cheapest model that works" has costs that don't appear in a token bill. The case for flagship-by-default is stronger than usually admitted — and there's a cheaper optimization people skip first.
-
Why Turn 20 Costs More Than Turn 2
A six-turn run whose final context is 12,000 tokens sends 42,000. That gap is agent economics — and it makes an early oversized tool result the most expensive thing in your run.
-
Graceful Degradation When the Model Provider Is Down
A fallback exercised only during incidents is exercised for the first time during an incident. Six degradation rungs, and the three cheapest ones cover most of the value.
-
Latency Budgets for Agents: Where the Seconds Actually Go
"Try a faster model" addresses maybe a third of the time in a typical run. Split the twelve seconds five ways and the real culprits — turn count and tool latency — are usually the ones nobody measured.
-
Queueing and Backpressure for Agent Workloads
A run can take four seconds or four minutes, and you can't tell in advance. Size the pool to the provider's rate limit, split queues by shape — and check the one timeout relationship that silently runs your task twice.
-
Agents That Wait: Callbacks, Webhooks, and Long-Running Tools
A twenty-minute export breaks the assumption that tools return quickly. Handles, in-loop polling, and true suspension — plus the expiry field that stops suspended runs leaking forever.
-
Fan-Out, Fan-In: Parallelism Inside a Single Agent
Twelve independent checks shouldn't cost twelve turns, each re-sending the whole context. Three levels of parallelism inside one agent — and the fan-in is where the pattern usually goes wrong.
-
Model Routing: Using the Cheap Model for the Cheap Steps
Classification, extraction, and formatting move to a cheaper tier safely. Tool selection and planning don't — and the failure looks like a tool-description bug, so you'll debug the wrong thing for a week.
-
Where You Put Things in the Context Changes the Answer
Same instructions, same documents, different results — because the constraint sat in the middle of a long array. Two ordering rules, one of which also decides whether your prompt cache ever hits.
-
Chatbot, Workflow, or Agent? Pick the Simplest Thing That Works
Six weeks building an agent for a task with three fixed steps. One question — can you draw the flowchart before seeing the input? — separates the three architectures and saves most of that time.