#observability
Every post tagged "observability".
22 posts · showing 11–20
-
Building an Eval Set From Production Traces
Hand-written cases test what you thought of. A user rephrasing the same request three times is a labelled failure sitting in your logs — here's how to turn a week of traces into a suite that catches the real ones.
-
Cost Attribution: Finding Which Feature Is Eating Your Token Budget
Agent spend is dominated by a tail of expensive runs, so every averaged chart hides the cause. Five tags on each model call, one table of your twenty priciest runs, and the mystery usually resolves in an afternoon.
-
The On-Call Agent: What It Should and Shouldn't Touch During an Incident
An incident is when your system is least understood — a poor moment for an autonomous actor, an excellent one for an autonomous investigator. The line between them is read access and write access.
-
Flaky Agents: Separating Model Nondeterminism From Your Bugs
A test passed on re-run. That's ambiguous for agents in a way it isn't for other software — and the fix is to stop reporting pass/fail and start reporting distributions.
-
Reading Your First Agent Trace: Wrong Tool, Wrong Argument, or Wrong Plan?
Four buckets, checked in order, and most agent failures resolve before you get to the interesting one. The trick is finding the first wrong turn and reading the raw array — not the pretty rendering.
-
Build or Buy the Eval Harness? What Each Choice Locks In
Teams buy an eval platform and keep their own scripts anyway. The reason is deterministic replay — it hooks into your dispatch layer, which is the one part no vendor can own.
-
Streaming an Agent's Progress Without Lying About It
Token streaming goes quiet exactly when an agent gets slow — inside the tool call. The fix is an event stream from the loop, and the hard part is not inventing certainty the loop doesn't have.
-
Chain, Blackboard, or Broadcast? Three Ways Agents Share State
Chains lose the evidence, blackboards fight over it, broadcasts drown in it. Pick by which failure your task can absorb — and log provenance from day one or debugging becomes guesswork.
-
Agent Adoption Stalls at Trust, Not Capability
An agent whose errors cluster somewhere describable beats a more accurate one whose errors are scattered. Ask your users when they double-check its work — the answer is your real roadmap.
-
Framework or Plain Loop? Four Questions That Actually Decide It
The core agent loop is under a hundred lines — that's never what a framework saves you. Four situational questions decide it, and the one about 3am debugging should carry the most weight.