Running in Production
Deployment, cost, latency, monitoring, scaling.
24 posts · showing 21–24
-
Cost Attribution: Finding Which Feature Is Eating Your Token Budget
Agent spend is dominated by a tail of expensive runs, so every averaged chart hides the cause. Five tags on each model call, one table of your twenty priciest runs, and the mystery usually resolves in an afternoon.
-
Rate Limits Are an Agent Design Constraint, Not an Ops Problem
Backoff assumes a predictable call count. Agents burst, retry at three layers, and generate more traffic the more they're throttled — which makes rate limits an architecture question, not an ops ticket.
-
Streaming an Agent's Progress Without Lying About It
Token streaming goes quiet exactly when an agent gets slow — inside the tool call. The fix is an event stream from the loop, and the hard part is not inventing certainty the loop doesn't have.
-
The Retry Storm Your Dashboard Can't See: When Agent Failures Multiply Instead of Add
Three retry layers, each sensible on its own, compose by multiplication — and a failure at step fourteen costs the whole context, not the failed call. Your per-call dashboard will stay green the entire time.