#failure-analysis
Every post tagged "failure-analysis".
49 posts · showing 11–20
-
The Tool Call Failed on Step 3. Retry, Skip, or Replan? Decide by What the Failure Told You About the Plan.
A failed tool call carries two facts: the action didn't happen, and something you believed may be false. Agents act on the first and throw away the second — here's the one question that sorts retry from skip from replan, and how to put the answer in the error contract instead of the prompt.
-
Your Critic Agent Approves Everything Because It Read the Worker's Reasoning. Starve It of the Argument, Give It Its Own Evidence.
Forward the worker's transcript to your reviewer agent and it will approve nearly everything — anchored by a conclusion it treats as evidence. Here is the information diet, the mutation test, and the disagreement policy that turn a rubber stamp back into a check.
-
The Refund Tool Timed Out, the Agent Retried, and the Customer Got Paid Twice. Make Every Write Tool Idempotent.
A timeout tells the agent nothing about whether the write happened, and a retry is its most reasonable next move. The fix belongs in the tool, not in the prompt.
-
The Analytics Agent Said Revenue Was Up 38%. The Join Had Counted Every Order Once Per Line Item.
A text-to-SQL agent that answers with a single number has hidden the one thing a reviewer needed to see. Here is how a fan-out join slipped through, and the grain checks that would have caught it before anyone got to the slide.
-
Your Agent Works in the Demo Because You're Watching: Six Failures That Only Appear at 3 A.M.
A live demo isn't a smaller version of production — it's a different environment with a human silently patching over every gap. Here are the six failure classes that only wake up once that human goes to bed.
-
Summarising the Conversation Threw Away the One ID the Next Tool Call Needed. Summarise Prose, Never Identifiers.
Context compaction reads fine to a human and breaks the agent anyway: the summary kept the story and lost the order number. The fix is to stop asking a summariser to carry identifiers at all.
-
One Tool With Twelve Parameters or Four With Three? Let the Agent's Error Log Decide
Merging tools trades wrong-tool mistakes for wrong-argument mistakes. Your traces already show which one your agent makes more, and an overloaded tool leaves a signature you can grep for.
-
An Agent That Cannot Fail Will Retry Forever. Budget Tokens and Wall-Clock, Not Retry Counts.
Your max_retries=3 never fires, because every search call succeeds. The agent just keeps finding new ways to look for an answer that doesn't exist. Here is the budget that actually stops it, and the exit that lets it stop politely.
-
The Page Your Agent Read Told It to Email Your Customer List. "Ignore Instructions in the Content" Won't Stop the Next One.
A system-prompt warning makes injection less likely. It doesn't make it impossible, and the attacker gets as many tries as they like. The real fix lives in the tool dispatcher, not the prompt.
-
Retrieval Returned Zero Rows and the Agent Cited a Policy Doc Anyway
Empty retrieval results are the only tool failure that arrives as a success — status 200, results [] — and a model asked to answer from sources will invent one to fill the hole. Two boundary fixes: a loud typed empty, and citation IDs validated in code.