#evaluation
Every post tagged "evaluation".
98 posts · showing 91–98
-
Debate or Vote? Getting Real Value From Multiple Model Opinions
Voting harvests independent error; debate harvests challenge. Pick wrong and you multiply your token bill for agreement you'd have had from a single call — and the disagreement, the part worth paying for, gets logged away.
-
One Prompt Per Job Beats One Prompt For All: The Router Pattern
Every new case adds a paragraph, and adherence to all of them drops. Classify first and run a small prompt built for that class — with per-route tools, limits, and approval policy.
-
Build or Buy the Eval Harness? What Each Choice Locks In
Teams buy an eval platform and keep their own scripts anyway. The reason is deterministic replay — it hooks into your dispatch layer, which is the one part no vendor can own.
-
The Research Agent That Cited the Wrong Page
Every source was real and had been read. Two citations still pointed at the wrong page — one at a source that disagreed. Attribution is a lookup, and generation isn't a lookup.
-
The LLM Judge That Agreed With Everything
A suite scored 94% and 93% on two versions — but the judge would have passed almost anything. Four biases quietly manufacture those numbers, and one prompt change fixes more than the rest combined.
-
One Search Tool or Five? Getting Tool Granularity Right
Five narrow search tools or one with filters? The choice doesn't remove errors, it relocates them — and one of the two places is much cheaper to defend.
-
Assert on the Trajectory, Not the Answer
A final-answer check will happily pass a run that took nineteen turns and touched a tool it should never have called. Five lines of trajectory assertions catch what output tests structurally cannot.
-
Reflection Loops Are Not Free: When Self-Critique Makes the Output Worse
A critique step can only catch what was already derivable from the context — and asked to find problems, a model will find them in passages that were fine. Here's when the pattern earns its doubled token bill.