Testing & Reliability
Evals, debugging, tracing, why agents fail.
30 posts · showing 21–30
-
How Many Eval Cases Do You Actually Need?
18/20 versus 19/20 is not an improvement — it's noise wearing a number. What your suite size can and cannot detect, and the paired design that makes a hundred cases behave like seven hundred.
-
Eval Cases Where the Right Answer Is "I Don't Know"
Suites measure whether the agent got it right, never whether it should have answered at all. Five case types where the correct response is a question — plus the paired cases that stop you from just making it timid.
-
Red-Teaming Your Own Agent: A Practical Injection Suite
Trying a few "ignore your instructions" prompts tests the easiest attack against the strongest defense. A real suite is a matrix of carriers, payloads, and targets — asserting on actions taken, not words said.
-
Bisecting a Bad Run: Finding the Change That Broke It
It worked last week. Six things could have changed, and they vary independently — so bisect across the dimensions first, then within one. And diff the raw arrays for what's missing, not what's different.
-
Testing Multi-Agent Systems Without Losing Your Mind
Add a second agent and end-to-end assertions stop telling you anything useful. Contracts at every seam, stubs for the supervisor's unhappy paths, and integration tests that assert on graph shape and cost.
-
Canary Releases for Prompt Changes
A sentence added to fix one behavior routinely changes three others. Ship prompts like behavior: versioned by rendered hash, split by stable key, and compared on tool distribution — the row nobody collects.
-
Building an Eval Set From Production Traces
Hand-written cases test what you thought of. A user rephrasing the same request three times is a labelled failure sitting in your logs — here's how to turn a week of traces into a suite that catches the real ones.
-
Flaky Agents: Separating Model Nondeterminism From Your Bugs
A test passed on re-run. That's ambiguous for agents in a way it isn't for other software — and the fix is to stop reporting pass/fail and start reporting distributions.
-
The LLM Judge That Agreed With Everything
A suite scored 94% and 93% on two versions — but the judge would have passed almost anything. Four biases quietly manufacture those numbers, and one prompt change fixes more than the rest combined.
-
Assert on the Trajectory, Not the Answer
A final-answer check will happily pass a run that took nineteen turns and touched a tool it should never have called. Five lines of trajectory assertions catch what output tests structurally cannot.