#evaluation
Every post tagged "evaluation".
98 posts · showing 11–20
-
How Should a Team Review a Tool Description Diff? Treat It Like a Schema Migration
A six-word edit to a tool description is a behavior change with no compiler to catch it. Borrowing the review checklist teams already use for schema migrations turns an unreviewable prose diff into something you can actually approve.
-
Your 95% Step Accuracy Is a 60% Agent
Compounding explains part of the gap between a green eval dashboard and a broken agent. The rest is that per-step scoring counts a step as correct when it succeeds on a premise that was already wrong.
-
Was It Retrieval or Reasoning? Diagnosing a Wrong Answer
A day spent rewriting the prompt for a wrong policy answer, when the policy document was never retrieved. Three checks in the right order, and one log field that separates "ignored it" from "never saw it."
-
The Screening Agent That Could Only Be Right
Every available metric said it was working — and every one of them was computed only from the candidates it advanced. The decision destroys the evidence needed to evaluate the decision.
-
Three Ways to Get Valid JSON, and What Each One Costs
The retry loop was working and quietly biased: the tickets needing three attempts weren't a random sample, and their eventual valid output was thinner than the truth.
-
Test the Tool or Test the Agent? What Each Layer Catches
Sort a month of eval failures by root cause and most turn out to be tool contract bugs — findable in milliseconds by a unit test, currently being caught by the slowest instrument you own.
-
When a Worker Returns Garbage: Validating at the Handoff
Thirty-nine extracts were fine. The fortieth had every field present, correctly typed, and null — and the supervisor aggregated it, because well-formed looked like correct.
-
What Ten Years of Agent Tooling Might Look Like
Specific capability predictions age badly; layer predictions age better. Which parts commoditize, which stay differentiated — and the one prediction here most likely to be wrong.
-
Positions Worth Revising About Agents
Eight beliefs that are reasonable on their face and don't survive production. Every correction points the same direction — which makes it a useful prior for the next problem.
-
Reading the Agent Discourse Without Being Misled
Two questions handle most of it: what specifically was the input, and what specifically checked the output? Almost every overstated claim is weak on one of them.