The LLM Judge That Agreed With Everything
A suite scored 94% and 93% on two versions — but the judge would have passed almost anything. Four biases quietly manufacture those numbers, and one prompt change fixes more than the rest combined.
An eval suite scored a new agent version at 94% and the previous one at 93%. Both numbers were produced by a model asked, for each output, "Is this response correct and helpful? Answer yes or no."
The judge would have scored almost anything in the low nineties. The question was built to be agreed with, and a model asked a yes-or-no question about plausible-looking text says yes. The suite wasn't measuring quality; it was measuring how agreeable the judge was, and reporting it to two significant figures.
Model-as-judge is a genuinely useful technique. It is also easy to build in a way that produces numbers with no relationship to what you care about, and the failure is invisible — the scores look fine, they move a little between versions, and nobody suspects anything until a version that scores well ships badly.
The four biases, and what each does to your numbers
Agreement bias. Yes/no questions about existing text skew heavily toward yes. Any judge prompt of the form "is this good?" inherits it.
Verbosity bias. Longer, more thorough-looking answers score higher regardless of correctness. This one actively harms you: it rewards exactly the hedging and padding you'd want an agent to avoid, so optimizing against the judge makes the product worse.
Position bias. In pairwise comparisons, one position — often the first — wins more often than it should, independent of content.
Self-preference. A judge tends to favour output from the same model family, phrased the way it would phrase it. Relevant whenever you're comparing across providers and the judge shares a lineage with one side.
Five fixes, in order of return
1. Ask for violations, not verdicts. Replace "is this correct?" with a specific extraction task:
Given the response and the requirements list, output every requirement
the response fails to satisfy. For each: the requirement, the reason,
and the quoted span. Output [] if none fail.
A judge that must quote evidence can't wave through a bad answer as easily, and the empty list is a meaningful signal rather than a default. This single change fixes more than the other four combined.
2. Give it a rubric with observable criteria. "Helpful" is unjudgeable. "Cites at least one retrieved document for each factual claim," "does not recommend an action outside the listed tools," "states the account tier" — each is checkable, and disagreements between two judges become resolvable rather than a matter of taste.
3. Randomize order and average both directions. For pairwise comparisons, run A-then-B and B-then-A, and count only agreements as decisive. Where the two orderings disagree, you've learned the judge can't tell them apart — which is information, not noise to be averaged away.
4. Control for length. Either constrain the outputs being compared to similar lengths, or include "conciseness is not a defect" in the rubric and check whether the judge's scores still correlate with length. If they do, your rubric hasn't overcome the bias.
5. Calibrate against human labels — and keep doing it. Hand-label a small set, fifty items is enough to start, and measure judge agreement with those labels. That agreement number is your judge's error bar. ⚠️ Without it, a 94% has no confidence interval and a one-point difference between versions is indistinguishable from noise.
🔍 A five-minute test of your existing judge
Take twenty items your judge scored as passing. Deliberately corrupt ten of them — introduce a factual error, drop a required field, add a plausible but wrong recommendation. Re-run the judge on all twenty, shuffled.
If it doesn't catch most of the corruptions, your scores measure nothing. Most first-draft judges catch a disappointing fraction, and finding that out costs almost nothing compared to shipping on the strength of the number.
What to do with the number afterwards
Judge scores are best used as a relative signal on a fixed suite, not as an absolute quality claim. Report them alongside the judge's human agreement rate, and treat differences smaller than that error bar as no difference at all. ✅ Pair every judge-scored run with deterministic assertions — schema validity, forbidden tools, required fields — since those are exact, free, and catch the failures a judge is worst at.
The takeaway
A model asked whether something looks good will say yes. Judges become useful when they're asked to find specific violations against explicit criteria, checked against human labels, and reported with the error bar that check produces. Until then, a rising score is as likely to reflect your outputs getting more agreeable as getting better.