#evaluation
Every post tagged "evaluation".
98 posts · showing 61–70
-
Agents Get Worse While You're Not Touching Them
Nobody edits anything and six months later it handles a whole class of request worse. Five things decay on their own — and none of them announce themselves by breaking.
-
From Prompt Edit to Signal in Under a Minute
A six-minute iteration buys you ten experiments a day; a thirty-second one buys a hundred. Four fixes, about an afternoon of work, and the seam that everything else depends on.
-
The Inbox Agent That Filed Everything Correctly and Lost Two Questions
Every filing decision was defensible, and two unanswered questions still vanished into the archive. The bug wasn't the prompt — it was that the agent judged messages while acting on conversations.
-
How Many Eval Cases Do You Actually Need?
18/20 versus 19/20 is not an improvement — it's noise wearing a number. What your suite size can and cannot detect, and the paired design that makes a hundred cases behave like seven hundred.
-
What I'd Tell a Team Starting Their First Agent Project
Seven things, none of them about model capability, and most of them cost a quarter before they're visible. The first one eliminates more candidate projects than anything else — and should be applied at the idea stage.
-
The QA Agent That Wrote Tests That Always Passed
Coverage climbed, the build stayed green, and a real regression sailed through. An agent asked for passing tests writes passing tests — and the easiest passing test verifies nothing.
-
Content Boundaries: What Your Agent Should Refuse, and How to Test It
Providers handle the general case. Scope, regulatory limits, commitments, and how your agent represents you are yours — and the test suite needs both halves, or you'll ship something that refuses everything.
-
Eval Cases Where the Right Answer Is "I Don't Know"
Suites measure whether the agent got it right, never whether it should have answered at all. Five case types where the correct response is a question — plus the paired cases that stop you from just making it timid.
-
Who Owns the Agent? The Team Structure Question
Prompts need domain expertise; tools need engineering; evals are specification. That split doesn't match team boundaries, which is why agent ownership goes ambiguous — and ambiguous ownership shows up as quality nobody watches.
-
How Many Agents Is Too Many?
Every addition is locally justified, and then the system costs more and answers worse than a simpler one would. Four measurable signals you've overshot — and the merge experiment that settles it.