All posts
Everything published here, newest first.
316 posts · showing 31–40
2026
10 posts-
Your Agent Works in the Demo Because You're Watching: Six Failures That Only Appear at 3 A.M.
A live demo isn't a smaller version of production — it's a different environment with a human silently patching over every gap. Here are the six failure classes that only wake up once that human goes to bed.
-
The Agent Got the Right Answer by the Wrong Route. Score the Trajectory, and Know the Three Cases Where the Answer Is Enough
A final-answer eval passes an agent that guessed, one that ran a write query mid-task, and one that took forty steps to do the job in four. Here is how to assert on the path without freezing a golden trace, and when you can safely skip it.
-
Summarising the Conversation Threw Away the One ID the Next Tool Call Needed. Summarise Prose, Never Identifiers.
Context compaction reads fine to a human and breaks the agent anyway: the summary kept the story and lost the order number. The fix is to stop asking a summariser to carry identifiers at all.
-
One Tool With Twelve Parameters or Four With Three? Let the Agent's Error Log Decide
Merging tools trades wrong-tool mistakes for wrong-argument mistakes. Your traces already show which one your agent makes more, and an overloaded tool leaves a signature you can grep for.
-
An Agent That Cannot Fail Will Retry Forever. Budget Tokens and Wall-Clock, Not Retry Counts.
Your max_retries=3 never fires, because every search call succeeds. The agent just keeps finding new ways to look for an answer that doesn't exist. Here is the budget that actually stops it, and the exit that lets it stop politely.
-
The Page Your Agent Read Told It to Email Your Customer List. "Ignore Instructions in the Content" Won't Stop the Next One.
A system-prompt warning makes injection less likely. It doesn't make it impossible, and the attacker gets as many tries as they like. The real fix lives in the tool dispatcher, not the prompt.
-
Retrieval Returned Zero Rows and the Agent Cited a Policy Doc Anyway
Empty retrieval results are the only tool failure that arrives as a success — status 200, results [] — and a model asked to answer from sources will invent one to fill the hole. Two boundary fixes: a loud typed empty, and citation IDs validated in code.
-
A Second Pass That Asks the Same Question Finds the Same Bugs. Change the Unit, Not the Temperature.
Raising the temperature on a review pass re-rolls the dice inside a frame that has already decided what is findable. Three properties determine what a pass can see, and a real second pass has to change at least two of them.
-
Forty Tools in One Prompt: What It Actually Costs, and the Three Signals That Say Split the Agent
Forty registered tools do not fail loudly — the agent just gets vague, picking near-synonyms and asking questions it had the answer to. Here are the three costs you're paying, and the signals in your traces that tell you to merge, gate, or finally split.
-
A 1% Flaky Tool Shouldn't Take You Down. Three Nested Retry Layers Will.
A tool that fails one call in a hundred is healthy. The retry logic wrapped around it is what turns a dependency's bad minute into a queue of hung user requests — and the load multiplier is at its worst exactly when the dependency is weakest.