#monitoring
Every post tagged "monitoring".
41 posts · showing 21–30
-
How Many Eval Cases Do You Actually Need?
18/20 versus 19/20 is not an improvement — it's noise wearing a number. What your suite size can and cannot detect, and the paired design that makes a hundred cases behave like seven hundred.
-
Running Agents on a Schedule: What Changes Without a User
An agent someone triggers has a safety property nobody designs for: a person is watching. Put it on a schedule and that goes away — along with the feedback loop that was catching the errors.
-
Tracing Standards for Agents: What Fits and What's Missing
A run is a trace and a turn is a span — the mapping is cleaner than expected. Two things break: payloads far too big for span attributes, and sampling that discards exactly the runs you need.
-
Who Owns the Agent? The Team Structure Question
Prompts need domain expertise; tools need engineering; evals are specification. That split doesn't match team boundaries, which is why agent ownership goes ambiguous — and ambiguous ownership shows up as quality nobody watches.
-
Graceful Degradation When the Model Provider Is Down
A fallback exercised only during incidents is exercised for the first time during an incident. Six degradation rungs, and the three cheapest ones cover most of the value.
-
Latency Budgets for Agents: Where the Seconds Actually Go
"Try a faster model" addresses maybe a third of the time in a typical run. Split the twelve seconds five ways and the real culprits — turn count and tool latency — are usually the ones nobody measured.
-
Audit Logs That Survive an Incident
Audit logs get designed for compliance and used during incidents. One question — who caused what, to whom, under what authority — decides every field, and most logs can't answer the authority part.
-
Pricing an Agent Product When Your Costs Are Variable
One customer can cost a hundred times another on the same plan, and they're often the ones getting the most value. Five pricing models, their failure modes, and the trap where every quality improvement cuts your margin.
-
Data Exfiltration Paths You Forgot Your Agent Had
A URL fetch sends data in the request. A rendered markdown image leaks from the viewer's browser, bypassing every tool-level control. Enumerate the paths — several of them look like features.
-
Deadlock, Livelock, and the Agent That Waits Forever
A deadlocked thread pool gives you a stack trace. Deadlocked agents produce fluent, plausible activity that looks like work — and bill you for it until someone notices.