#reliability
Every post tagged "reliability".
124 posts · showing 111–120
-
The Demo-to-Production Gap Is an Evaluation Gap
Demos answer "can it?" Deployments ask "how often, across everything real users send, when the tools misbehave?" Five variants of your demo scenario will tell you more than the demo did.
-
The Agent That Read Page 1 of 40: Pagination in Tool Design
It reported three overdue invoices. There were sixty-one. The tool worked as documented — which is exactly why this bug is so common and so quiet.
-
Designing for Resumability: Agents That Survive a Restart
Serializing the message array is the obvious approach and it produces subtly wrong resumptions. What has to survive a crash is a work ledger, idempotent side effects, and enough state to rebuild context from scratch.
-
Flaky Agents: Separating Model Nondeterminism From Your Bugs
A test passed on re-run. That's ambiguous for agents in a way it isn't for other software — and the fix is to stop reporting pass/fail and start reporting distributions.
-
The Research Agent That Cited the Wrong Page
Every source was real and had been read. Two citations still pointed at the wrong page — one at a source that disagreed. Attribution is a lookup, and generation isn't a lookup.
-
The LLM Judge That Agreed With Everything
A suite scored 94% and 93% on two versions — but the judge would have passed almost anything. Four biases quietly manufacture those numbers, and one prompt change fixes more than the rest combined.
-
What Belongs in Long-Term Memory (and What's Just Transcript)
Embedding every turn gives you a transcript with a search box, not memory. Three categories deserve to outlive a conversation — and the contradiction problem needs deciding before it arrives, not after.
-
Prompt, Tool, or Code? Where Each Piece of Logic Belongs
A rule the agent keeps breaking usually isn't a prompting failure — it's deterministic logic filed under "advice." Two questions place any behavior in the right one of three homes.
-
Assert on the Trajectory, Not the Answer
A final-answer check will happily pass a run that took nineteen turns and touched a tool it should never have called. Five lines of trajectory assertions catch what output tests structurally cannot.
-
Trim, Summarize, or Retrieve? Three Ways to Survive a Long Agent Run
Trimming loses the oldest information, summarizing loses specificity, retrieval loses whatever the query missed. Which loss your run can survive is the whole decision — plus one rule that prevents most long-run failures.