#testing
Every post tagged "testing".
19 posts · showing 1–10
-
How Many Eval Cases Do You Actually Need?
18/20 versus 19/20 is not an improvement — it's noise wearing a number. What your suite size can and cannot detect, and the paired design that makes a hundred cases behave like seven hundred.
-
The QA Agent That Wrote Tests That Always Passed
Coverage climbed, the build stayed green, and a real regression sailed through. An agent asked for passing tests writes passing tests — and the easiest passing test verifies nothing.
-
Content Boundaries: What Your Agent Should Refuse, and How to Test It
Providers handle the general case. Scope, regulatory limits, commitments, and how your agent represents you are yours — and the test suite needs both halves, or you'll ship something that refuses everything.
-
Eval Cases Where the Right Answer Is "I Don't Know"
Suites measure whether the agent got it right, never whether it should have answered at all. Five case types where the correct response is a question — plus the paired cases that stop you from just making it timid.
-
Retry Semantics: Which Tools Are Safe to Call Twice
Agents repeat tool calls for at least five ordinary reasons — none of them bugs. Whether that's harmless or a duplicate charge is a property of the tool, and usually one nobody wrote down.
-
Red-Teaming Your Own Agent: A Practical Injection Suite
Trying a few "ignore your instructions" prompts tests the easiest attack against the strongest defense. A real suite is a matrix of carriers, payloads, and targets — asserting on actions taken, not words said.
-
Bisecting a Bad Run: Finding the Change That Broke It
It worked last week. Six things could have changed, and they vary independently — so bisect across the dimensions first, then within one. And diff the raw arrays for what's missing, not what's different.
-
Treating Prompts Like Code: Versioning, Review, and Rollback
A one-line edit in a web console changes behavior across every request, with no history and no way back. Prompts in the repo, rendered hashes in the logs, eval results in the PR — and a quarterly prune.
-
Testing Multi-Agent Systems Without Losing Your Mind
Add a second agent and end-to-end assertions stop telling you anything useful. Contracts at every seam, stubs for the supervisor's unhappy paths, and integration tests that assert on graph shape and cost.
-
The Verifier in the Loop: Designing an Oracle Your Agent Can Iterate Against
Coding agents work as well as they do because compilers exist. In other domains the oracle is something you build — and most tasks have one hiding in them if you spend ten minutes looking.