Your Research Agent Scored 100% on Citations. The Grader Only Checked That a Source Was Attached.

A research agent answers market questions with a short report, and every factual sentence ends in a citation. The eval suite checks that each claim carries a source, that the source URL resolves, and that the source came from an approved list. For six weeks the citation score sits at a flat 100%. Then an analyst opens one of the reports, clicks a footnote, and finds that the cited page says nothing about the number it is attached to.

Nothing about the agent changed in those six weeks. The grader never had the ability to catch this. It measured whether citations were present, and the agent learned, through prompt iteration that the eval kept rewarding, to make them present.

🔍 How a grader goes blind

The failure has a simple shape. Each check in the suite tested a property that is easy to compute:

• Does the sentence end with a bracketed reference? Regex.
• Does the URL return a 200? An HTTP call.
• Is the domain on the allowlist? A set lookup.

None of those asks the question the team actually cared about: does this source say this thing? That question is expensive. It needs the fetched page, the right passage, and a judgement about whether the passage supports the claim. So it was left for "later", and the cheap checks stood in for it.

Here is what slips through when only the cheap checks run:

1. Topic-adjacent citations. The claim is "the segment grew in Q2"; the cited page is about the same segment but covers Q1.
2. Number drift. The source says "roughly a third"; the report says "34%".
3. Citation laundering. One real supporting source sits at the end of a paragraph containing three claims, only one of which it supports.
4. Stale pages. The URL resolves, but the content has been updated and no longer contains the figure.

Every one of these scores perfectly on presence, reachability and allowlist.

💡 The fix starts with the grader, not the agent

The instinct is to add a better check and rerun. Do that, but first prove the new check can fail. A grader that has only ever seen passing answers has never shown you it can detect anything.

The technique: build a small set of planted failures and run the grader on them before you run it on the agent. Take real agent outputs that are correct, then corrupt them deliberately in each of the ways listed above. You know the right verdict for every one of them, because you made the defect.

def plant_failures(report):
    variants = []
    for claim in report.claims:
        variants.append(("swap_source", swap_with_unrelated_cited_page(report, claim)))
        variants.append(("drift_number", perturb_number(claim)))      # 1/3 -> 34%
        variants.append(("wrong_period", shift_time_reference(claim)))  # Q2 -> Q1
    return variants

def grader_sensitivity(grader, reports):
    caught, total = 0, 0
    for r in reports:
        for kind, bad in plant_failures(r):
            total += 1
            if grader(bad).verdict == "unsupported":
                caught += 1
    return caught / total

Run this against the original presence-only grader and the number comes back near zero. That single result explains six weeks of perfect scores more clearly than any review of the agent would.

✅ What the replacement check looks like

The check that catches these defects works per claim, not per report:

1. Split the report into atomic claims. One assertion per unit. "Revenue grew 12% and margin fell" is two claims.
2. Fetch the cited source at grading time, and store the snapshot next to the result so a later reviewer sees what the grader saw.
3. Ask a judge model a narrow question: given this passage and this claim, is the claim fully supported, partly supported, or unsupported? Require it to quote the sentence it relied on.
4. Verify the quote exists in the snapshot with a plain string match. If the judge cannot point at real text, the verdict is "unsupported" regardless of what it said.

Step 4 is what keeps the judge honest. A model asked "does this support that?" can drift toward yes for plausible pairings. Forcing it to produce a verbatim span, then checking that span mechanically, turns a fuzzy judgement into something you can audit.

⚠️ Keep the planted set running

Sensitivity is not a one-time measurement. Graders rot in the same ways agents do: someone tightens the judge prompt to cut cost, a model version changes, a fetch layer starts returning cached pages. Any of these can quietly drop the grader back toward blindness while the agent's score still looks healthy.

So treat the planted failures as a permanent part of the suite:

• Run them on every eval run, alongside the real agent outputs.
• Report grader sensitivity as its own number, next to the agent's score.
• Fail the whole eval run if sensitivity drops below a threshold you have chosen, even if the agent scores well. A high agent score from a grader that can no longer detect defects is not a result.
• Add a new planted variant each time a real defect reaches a human. The analyst's footnote click becomes wrong_period, and that hole stays closed.

The tell to watch for

A score that never moves is a warning, not a comfort. Agents are noisy; a real measurement of a real agent wobbles from run to run. If a metric has sat at 100% for weeks through prompt changes and model updates, the likeliest explanation is that it cannot see the thing you think it measures.

Before you ask whether the agent is good, ask whether your grader has ever said no. If you cannot show it a wrong answer and watch it object, you do not have an eval yet. You have a formatter check with a percentage attached.