#evaluation
Every post tagged "evaluation".
35 posts · showing 31–35
-
The Research Agent That Cited the Wrong Page
Every source was real and had been read. Two citations still pointed at the wrong page — one at a source that disagreed. Attribution is a lookup, and generation isn't a lookup.
-
The LLM Judge That Agreed With Everything
A suite scored 94% and 93% on two versions — but the judge would have passed almost anything. Four biases quietly manufacture those numbers, and one prompt change fixes more than the rest combined.
-
One Search Tool or Five? Getting Tool Granularity Right
Five narrow search tools or one with filters? The choice doesn't remove errors, it relocates them — and one of the two places is much cheaper to defend.
-
Assert on the Trajectory, Not the Answer
A final-answer check will happily pass a run that took nineteen turns and touched a tool it should never have called. Five lines of trajectory assertions catch what output tests structurally cannot.
-
Reflection Loops Are Not Free: When Self-Critique Makes the Output Worse
A critique step can only catch what was already derivable from the context — and asked to find problems, a model will find them in passages that were fine. Here's when the pattern earns its doubled token bill.