#observability
Every post tagged "observability".
40 posts · showing 1–10
-
Before You Adopt an Agent Framework, Try to Break Its Loop: Four Escape Hatches You'll Need by Week Three
Time-to-demo is the one framework criterion that stops mattering the moment you ship. Four seams to test in an afternoon — and the production incident each one predicts.
-
The Agent Got the Right Answer by the Wrong Route. Score the Trajectory, and Know the Three Cases Where the Answer Is Enough
A final-answer eval passes an agent that guessed, one that ran a write query mid-task, and one that took forty steps to do the job in four. Here is how to assert on the path without freezing a golden trace, and when you can safely skip it.
-
A 1% Flaky Tool Shouldn't Take You Down. Three Nested Retry Layers Will.
A tool that fails one call in a hundred is healthy. The retry logic wrapped around it is what turns a dependency's bad minute into a queue of hung user requests — and the load multiplier is at its worst exactly when the dependency is weakest.
-
Can You See the Raw Messages? The Question That Sorts Agent Frameworks
Every agent framework demos well. The one that survives your first 2am incident is the one that will hand you the exact bytes it sent to the model — and let you change them before they go.
-
One 300-Second Timeout Tells You Nothing. Agents Need a Deadline Budget.
A run that dies at its ceiling with no attribution tells you nothing about which part was slow — and pushes you toward the laziest fix, raising the ceiling. Three nested clocks, an inherited remainder, and a reserved tail slice turn timeouts into a signal you can actually read.
-
Retry, Repair, or Replan? A Recovery Ladder for When an Agent's Step Fails
Blind retry burns money on a call that will never succeed; premature replanning throws away correct work and produces a worse plan. The deciding signal isn't how bad the error looks — it's what the error proves false.
-
Framework or Raw SDK? The Test Is Whether You Can Print the Exact Prompt
The framework debate is usually argued on taste. There's a faster test: build a two-tool agent and try to dump the exact request it sends on turn four. How hard that is predicts your next six months of debugging.
-
Your Eval Ran Each Case Once. That's Why It Missed the Flake.
A forty-case suite can pass every case and still hide a bug that reaches users one week in five. The number missing from most agent evals isn't coverage — it's how many times you ran each case.
-
Green Evals, Broken Production: The Frozen Fixture Problem
An eval suite built on recorded tool responses stops testing your agent and starts testing a museum. Here are three cheap detectors for fixture rot — and the rule for when a case should call the live tool instead.
-
Your 95% Step Accuracy Is a 60% Agent
Compounding explains part of the gap between a green eval dashboard and a broken agent. The rest is that per-step scoring counts a step as correct when it succeeds on a premise that was already wrong.