#failure-analysis
Every post tagged "failure-analysis".
49 posts · showing 1–10
-
The Analytics Agent Answered "What Was Q3 Revenue?" From a Deprecated Table. Make It Name Its Source Before Its Number.
The SQL was valid, the arithmetic was right, and the number was still wrong, because the agent picked the table whose name looked most like the question. Here is how to make a data agent commit to a source you can check before it hands anyone a figure.
-
The Invoice Tool Timed Out, So the Billing Agent Called It Again. The Customer Was Charged Twice. A Timeout Is Not a Failure.
A timeout tells the agent it stopped waiting, not that nothing happened. Here is how a reasonable-looking retry turned into a double charge, and the idempotency key that makes the retry safe.
-
Your Research Agent Scored 100% on Citations. The Grader Only Checked That a Source Was Attached.
An eval that has never failed anything is not proof of a good agent. Before you trust a grader's score, feed it answers you know are wrong and see if it notices.
-
The Travel Agent Reviewed Its Itinerary Three Times and Approved the Wrong Date Every Time. Reflection Needs Something to Check Against.
A self-review loop that reads only its own draft can polish the wrong answer forever. The fix is to give the critic something the generator never had: the original request, a fresh context, or a tool that can say no.
-
The Expense Agent Logged "Rate limit exceeded" as a Vendor Name. Make Tool Errors Impossible to Mistake for Results.
When a tool returns its failure as a plain string, the model reads it like any other output and carries on. The fix lives in the tool's return shape, not in the prompt.
-
The Refund Tool Capped Each Refund at $500. The Support Agent Issued Eleven $490 Refunds to One Customer.
A per-call limit only answers "is this action too big?" Agents fail on a different question: "is this sequence too big?" Here is how a guardrail that never fired still let the money leave, and the ledger check that would have stopped it.
-
Your Scheduling Agent Moved One Meeting and Said "Done." Teach the Loop What Finished Looks Like.
An agent loop ends when the model stops asking for tools, not when your task is finished. Here is why that gap produces confident half-done work, and a small done-check that closes it.
-
The On-Call Agent Restarted the Primary Anyway. The Instruction Not To Had Been Summarised Away.
Context compaction keeps the story of a long session and quietly drops the rules. The fix is a pinned constraint block that the summariser is never allowed to touch, plus a check that proves it survived.
-
The Inbox Agent Said Nobody Had Replied. The Search Tool Had Returned Page One of Four.
An agent can't tell "there's nothing here" from "I only saw the first page" unless the tool tells it. Here is how to make a paginated tool announce its own incompleteness, and why claims of absence need a stricter rule than claims of presence.
-
Your Agent Fleet Kept Its Own Quota Pinned for Forty Minutes: Synchronised Retries Are a Self-Inflicted Outage
The provider rejected one batch of calls; the fleet turned that into forty minutes of self-inflicted downtime. Why exponential backoff alone makes synchronised retries worse, the sub-second signal that exposes it, and the three-part fix that actually breaks the loop.