#testing
Every post tagged "testing".
43 posts · showing 31–40
-
Bisecting a Bad Run: Finding the Change That Broke It
It worked last week. Six things could have changed, and they vary independently — so bisect across the dimensions first, then within one. And diff the raw arrays for what's missing, not what's different.
-
Treating Prompts Like Code: Versioning, Review, and Rollback
A one-line edit in a web console changes behavior across every request, with no history and no way back. Prompts in the repo, rendered hashes in the logs, eval results in the PR — and a quarterly prune.
-
Testing Multi-Agent Systems Without Losing Your Mind
Add a second agent and end-to-end assertions stop telling you anything useful. Contracts at every seam, stubs for the supervisor's unhappy paths, and integration tests that assert on graph shape and cost.
-
The Verifier in the Loop: Designing an Oracle Your Agent Can Iterate Against
Coding agents work as well as they do because compilers exist. In other domains the oracle is something you build — and most tasks have one hiding in them if you spend ten minutes looking.
-
Canary Releases for Prompt Changes
A sentence added to fix one behavior routinely changes three others. Ship prompts like behavior: versioned by rendered hash, split by stable key, and compared on tool distribution — the row nobody collects.
-
Why Two Identical Runs Give Different Answers
Same input, different result. Four things cause it, only one is a bug — and the most common culprit is that the input wasn't actually identical.
-
Migrating Model Versions Without Breaking Your Agent
The new model is better at nearly everything and different at everything, and tool selection is what shifts first. Replay a frozen eval set against both — and put the cost-per-task row in the comparison.
-
Building an Eval Set From Production Traces
Hand-written cases test what you thought of. A user rephrasing the same request three times is a labelled failure sitting in your logs — here's how to turn a week of traces into a suite that catches the real ones.
-
The Demo-to-Production Gap Is an Evaluation Gap
Demos answer "can it?" Deployments ask "how often, across everything real users send, when the tools misbehave?" Five variants of your demo scenario will tell you more than the demo did.
-
Flaky Agents: Separating Model Nondeterminism From Your Bugs
A test passed on re-run. That's ambiguous for agents in a way it isn't for other software — and the fix is to stop reporting pass/fail and start reporting distributions.