The Documentation Agent That Actually Kept Docs in Sync

Documentation drifts because updating it is a separate task from the change that invalidated it, and separate tasks lose. An agent watching code changes and proposing doc updates is an obvious application — and the obvious implementation produces a stream of low-value pull requests that get closed unread.

What separated a working version from that one was mostly about what triggers it and what it's allowed to change.

The failure of the naive version

Point an agent at every merged PR and ask "does this change the docs?" and it says yes far too often, because almost any change could warrant a doc note. You get proposals to document internal refactors, to add explanatory paragraphs nobody asked for, and to restate in prose what the code already says.

The reviewers who must approve these are the same people who didn't want to write docs. They close them, and the agent's proposals become noise on every PR.

Trigger on contract changes, not on code changes

The version that worked ignored most commits and watched for changes to things other people depend on:

• Public function and endpoint signatures — parameters, return shapes, error codes.
• Configuration keys and their defaults.
• CLI flags and commands.
• Schema and migration files.
• Anything already referenced by an existing doc page.

Detecting these is mechanical — a diff filter, not a judgment call — which means the model isn't deciding whether a change is documentation-relevant. It's only deciding what the doc should say once something already established that it must change.

→ That last category is the highest-signal trigger: if a doc page references a symbol and that symbol changed, the page is stale by construction. No judgment required, no false positives.

Constrain what it may change

Proposals stayed reviewable because the agent's remit was bounded:

• ✅ Update statements about things that changed. A parameter's name, a default value, an error condition.
• ✅ Flag contradictions. "This page says the default is 30; the code now says 60." Even without proposing wording, the flag is valuable.
• ✅ Mark pages stale. For something it can't fix confidently, add a note rather than guessing.
• ❌ No new prose. No added explanation, no expanded examples, no restructuring. Those are human decisions with taste involved, and it's where the noise came from.

The reviewer's job became verifying a factual correction, which takes seconds, rather than evaluating writing, which takes minutes and invites disagreement.

Cite the source of every change

Each proposed edit carries the commit and the specific lines that made it necessary:

docs/api/orders.md line 44
  - "Returns up to 20 orders by default"
  + "Returns up to 50 orders by default"
  source: src/api/orders.py:112 (PR #4471) — DEFAULT_LIMIT 20 → 50

This is what makes review fast: the reviewer checks a claim against a diff, rather than trusting a summary. It also means a wrong proposal is obviously wrong, which is the property that keeps trust intact.

Where the real value turned out to be

Not the automatic updates — the contradiction reports.

Running the same check across the whole documentation set, rather than only on changed files, surfaced pages that had been wrong for months: defaults that changed a year ago, endpoints documented with parameters that no longer existed, examples referencing removed flags.

That backlog was invisible because nobody re-reads documentation they aren't currently changing. ⚠️ A one-off sweep like this is often more valuable than the ongoing automation, and it's a good first project — it produces a concrete list on day one instead of a pipeline whose value accrues slowly.

Measuring it

• Merge rate on proposals. Below a threshold, the agent is proposing things reviewers don't want; tighten the trigger.
• Time to review. Should be under a minute. Longer means the proposals are too broad.
• Doc-related bug reports over time. The actual outcome measure, and the one that justifies keeping it running.

The takeaway

A documentation agent works when it's triggered by contract changes rather than by any change, and restricted to correcting statements that are now factually wrong. Cite the source line for every edit so review is verification instead of judgment. And run the contradiction check across the whole corpus once — the backlog it finds is usually worth more than everything the automation will catch in its first year.