#monitoring
Every post tagged "monitoring".
41 posts · showing 1–10
-
Your Agent Fleet Kept Its Own Quota Pinned for Forty Minutes: Synchronised Retries Are a Self-Inflicted Outage
The provider rejected one batch of calls; the fleet turned that into forty minutes of self-inflicted downtime. Why exponential backoff alone makes synchronised retries worse, the sub-second signal that exposes it, and the three-part fix that actually breaks the loop.
-
Your Agent Works in the Demo Because You're Watching: Six Failures That Only Appear at 3 A.M.
A live demo isn't a smaller version of production — it's a different environment with a human silently patching over every gap. Here are the six failure classes that only wake up once that human goes to bed.
-
A 5% Canary Mostly Samples Your Easy Tasks. Roll Agent Upgrades Out by Task Class.
Random traffic slices sample volume, but an agent's risk lives in the thin tail of long, tool-heavy, irreversible tasks. Here's how to build a rollout ladder that samples the danger instead.
-
One 300-Second Timeout Tells You Nothing. Agents Need a Deadline Budget.
A run that dies at its ceiling with no attribution tells you nothing about which part was slow — and pushes you toward the laziest fix, raising the ceiling. Three nested clocks, an inherited remainder, and a reserved tail slice turn timeouts into a signal you can actually read.
-
One Worker Pool or Two? Separating Fast Agent Runs From Slow Ones
A handful of long research runs can starve hundreds of eight-second ticket replies without a single error firing. The fix isn't more workers — it's deciding, at admission, which lane a run belongs in.
-
Your Agents Asked 40 Questions This Week. Which Ones Should Have Reached a Human?
Human attention is the one resource in an agent deployment that never shows up on the bill — so agents spend it freely. Here's a three-way routing rule, a threshold you can actually tune, and the two metrics that tell you when your agent is asking too much or too little.
-
The First Hour of an Agent Incident
Nothing crashed, no alert fired, and a customer just forwarded a policy answer your support agent invented. The hardest question in the first hour isn't what broke — it's when it started.
-
Your Trace Store Will Outgrow Your Model Bill
Storage passed inference cost in three weeks, and the proposed fix — sample at 5% — would have discarded exactly the runs worth keeping. A trace isn't one artifact with one lifetime.
-
When to Stop Improving an Agent
There's always another failing case, so without a stopping rule the work either never ends or ends arbitrarily. The signal isn't the error rate — it's effort per point of improvement.
-
The Economics of an Agent-Heavy Team
Several line items move and only one of them is tokens. The savings are real, land on a single term, and are partly consumed by four others that grow — which is why big speedups produce modest delivery gains.