Running in Production
Deployment, cost, latency, monitoring, scaling.
24 posts · showing 11–20
-
Your Trace Store Will Outgrow Your Model Bill
Storage passed inference cost in three weeks, and the proposed fix — sample at 5% — would have discarded exactly the runs worth keeping. A trace isn't one artifact with one lifetime.
-
Agent Operations: The Role Without a Title Yet
Someone in every agent-running organization does this job without being hired for it. It has no name, no allocation, and no incident that forces it — until whoever was doing it informally leaves.
-
The Deploy Step Nobody Should Automate Yet
It's the easiest step to automate, saves the least time, and carries the largest tail risk. Automate the readiness report and the rollback instead — and separate deploying from releasing.
-
Which Agent Alerts Should Wake Someone Up?
An agent can be green on every service metric while doing the wrong work at ten times the cost. One rule sorts the alerts — and the highest-value one is the one almost nobody has configured.
-
Running Agents on a Schedule: What Changes Without a User
An agent someone triggers has a safety property nobody designs for: a person is watching. Put it on a schedule and that goes away — along with the feedback loop that was catching the errors.
-
Graceful Degradation When the Model Provider Is Down
A fallback exercised only during incidents is exercised for the first time during an incident. Six degradation rungs, and the three cheapest ones cover most of the value.
-
Latency Budgets for Agents: Where the Seconds Actually Go
"Try a faster model" addresses maybe a third of the time in a typical run. Split the twelve seconds five ways and the real culprits — turn count and tool latency — are usually the ones nobody measured.
-
Multi-Tenancy for Agent Products: The Boundaries That Must Hold
Data, memory, context, and capacity — four boundaries, and two of them are agent-specific enough that no existing multi-tenancy checklist mentions them. Few-shot examples harvested from production are a leak.
-
Queueing and Backpressure for Agent Workloads
A run can take four seconds or four minutes, and you can't tell in advance. Size the pool to the provider's rate limit, split queues by shape — and check the one timeout relationship that silently runs your task twice.
-
Migrating Model Versions Without Breaking Your Agent
The new model is better at nearly everything and different at everything, and tool selection is what shifts first. Replay a frozen eval set against both — and put the cost-per-task row in the comparison.