#evaluation
Every post tagged "evaluation".
35 posts · showing 1–10
-
How Many Eval Cases Do You Actually Need?
18/20 versus 19/20 is not an improvement — it's noise wearing a number. What your suite size can and cannot detect, and the paired design that makes a hundred cases behave like seven hundred.
-
What I'd Tell a Team Starting Their First Agent Project
Seven things, none of them about model capability, and most of them cost a quarter before they're visible. The first one eliminates more candidate projects than anything else — and should be applied at the idea stage.
-
The QA Agent That Wrote Tests That Always Passed
Coverage climbed, the build stayed green, and a real regression sailed through. An agent asked for passing tests writes passing tests — and the easiest passing test verifies nothing.
-
Content Boundaries: What Your Agent Should Refuse, and How to Test It
Providers handle the general case. Scope, regulatory limits, commitments, and how your agent represents you are yours — and the test suite needs both halves, or you'll ship something that refuses everything.
-
Eval Cases Where the Right Answer Is "I Don't Know"
Suites measure whether the agent got it right, never whether it should have answered at all. Five case types where the correct response is a question — plus the paired cases that stop you from just making it timid.
-
Who Owns the Agent? The Team Structure Question
Prompts need domain expertise; tools need engineering; evals are specification. That split doesn't match team boundaries, which is why agent ownership goes ambiguous — and ambiguous ownership shows up as quality nobody watches.
-
How Many Agents Is Too Many?
Every addition is locally justified, and then the system costs more and answers worse than a simpler one would. Four measurable signals you've overshot — and the merge experiment that settles it.
-
The Sales-Research Agent and the Problem of Stale Facts
Every fact in the brief had been true. None was labelled with when — so someone opened a call congratulating a person on a role they'd left a year ago.
-
Red-Teaming Your Own Agent: A Practical Injection Suite
Trying a few "ignore your instructions" prompts tests the easiest attack against the strongest defense. A real suite is a matrix of carriers, payloads, and targets — asserting on actions taken, not words said.
-
When Memory Makes an Agent Worse
Every retrieved memory is content injected into the context, relevant or not. Three ways that degrades the agent — including the one where it argues with a user about their own stated preference.