#testing
Every post tagged "testing".
19 posts · showing 11–19
-
Canary Releases for Prompt Changes
A sentence added to fix one behavior routinely changes three others. Ship prompts like behavior: versioned by rendered hash, split by stable key, and compared on tool distribution — the row nobody collects.
-
Why Two Identical Runs Give Different Answers
Same input, different result. Four things cause it, only one is a bug — and the most common culprit is that the input wasn't actually identical.
-
Migrating Model Versions Without Breaking Your Agent
The new model is better at nearly everything and different at everything, and tool selection is what shifts first. Replay a frozen eval set against both — and put the cost-per-task row in the comparison.
-
Building an Eval Set From Production Traces
Hand-written cases test what you thought of. A user rephrasing the same request three times is a labelled failure sitting in your logs — here's how to turn a week of traces into a suite that catches the real ones.
-
The Demo-to-Production Gap Is an Evaluation Gap
Demos answer "can it?" Deployments ask "how often, across everything real users send, when the tools misbehave?" Five variants of your demo scenario will tell you more than the demo did.
-
Flaky Agents: Separating Model Nondeterminism From Your Bugs
A test passed on re-run. That's ambiguous for agents in a way it isn't for other software — and the fix is to stop reporting pass/fail and start reporting distributions.
-
Build or Buy the Eval Harness? What Each Choice Locks In
Teams buy an eval platform and keep their own scripts anyway. The reason is deterministic replay — it hooks into your dispatch layer, which is the one part no vendor can own.
-
The LLM Judge That Agreed With Everything
A suite scored 94% and 93% on two versions — but the judge would have passed almost anything. Four biases quietly manufacture those numbers, and one prompt change fixes more than the rest combined.
-
Assert on the Trajectory, Not the Answer
A final-answer check will happily pass a run that took nineteen turns and touched a tool it should never have called. Five lines of trajectory assertions catch what output tests structurally cannot.