#failure-analysis
Every post tagged "failure-analysis".
49 posts · showing 21–30
-
A 1% Flaky Tool Shouldn't Take You Down. Three Nested Retry Layers Will.
A tool that fails one call in a hundred is healthy. The retry logic wrapped around it is what turns a dependency's bad minute into a queue of hung user requests — and the load multiplier is at its worst exactly when the dependency is weakest.
-
Your Agent Called the Same Tool Three Times in a Row. Here's What the Loop Is Telling You.
A repeating agent isn't being stubborn — it's being consistent. Identical tool calls mean the context barely changed between turns, and there are only four reasons that happens.
-
Retry, Repair, or Replan? A Recovery Ladder for When an Agent's Step Fails
Blind retry burns money on a call that will never succeed; premature replanning throws away correct work and produces a worse plan. The deciding signal isn't how bad the error looks — it's what the error proves false.
-
Your Eval Ran Each Case Once. That's Why It Missed the Flake.
A forty-case suite can pass every case and still hide a bug that reaches users one week in five. The number missing from most agent evals isn't coverage — it's how many times you ran each case.
-
Your Agent Retried the Tool Call. The Customer Got Charged Twice.
A timed-out tool call is not a failed one — it is an unknown one. Here is why "just retry" quietly doubles real-world side effects, and where to put the idempotency check so it cannot.
-
Your Agent Has One manage_calendar Tool. That's Why It Deleted the Wrong Meeting.
A single do-everything tool looks tidy in the schema and hides four separate problems: guessed actions, unvalidated payloads, permissions that can't tell reading from deleting, and traces that flatten three risk levels into one name. Here's where to cut instead — and where splitting makes things worse.
-
Green Evals, Broken Production: The Frozen Fixture Problem
An eval suite built on recorded tool responses stops testing your agent and starts testing a museum. Here are three cheap detectors for fixture rot — and the rule for when a case should call the live tool instead.
-
When Memory Contradicts the Tool Call, Which One Does the Agent Believe?
An agent had the live billing data and a three-week-old memory in the same context window, and answered from the memory. The reason isn't retrieval — it's how prompts are assembled, and there are three specific fixes.
-
The Tool Returned 200 and Nothing Happened
A calendar agent reported success on a meeting that landed at 3am, on the wrong calendar, with no invitation sent. The API did exactly what it was asked — and the return value gave the agent no way to know.
-
Read-Only Was Never the Same as Safe
SELECT-only credentials close exactly one of the four ways an analytics agent can hurt you. The other three don't require the ability to write anything to the database at all.