How Should a Team Review a Tool Description Diff? Treat It Like a Schema Migration

A pull request changes six words in a tool description. Search the knowledge base. becomes Search the knowledge base for policy documents. Two approvals, merged, no tests to run. The following week the support agent stops calling that tool for anything that isn't obviously policy-shaped, and a chunk of its answers quietly get worse.

Nothing in that diff was wrong. The reviewers read the sentence, agreed it was clearer, and shipped it. The mistake was reviewing it as prose. A tool description is an interface contract whose consumer is a probability distribution — and the closest thing most teams already know how to review carefully is a database migration.

🔌 Why tool descriptions slip past review

Code review works because meaning is local and mechanical. A reviewer can trace a changed function to its callers, and if they miss something, the type checker or the test suite catches it.

A tool description has neither property:

• The callers are invisible. Nothing in the repo references the description string except the schema it lives in. Grep tells you nothing about who depends on the phrase "for policy documents."
• Nothing fails loudly. A worse description doesn't throw. The agent just picks a different tool, or the same tool less often, or fills an argument differently. The failure shows up as a drift in answer quality, days later, attributed to "the model."
• The change is scoped to a sentence but lands on every request. Every call the agent will ever make gets routed through that text. It is one of the highest-fan-out lines in the system, and it looks like a docstring.

The same is true of the enum values, the parameter names, and the one-line argument descriptions inside the JSON schema. date versus as_of_date versus report_date is not cosmetic — it is the only signal the model has about what to put there.

🏗️ Borrow the checklist from schema migrations

Teams that would wave through a prose diff will happily spend twenty minutes on a migration PR, because migrations have a well-known review ritual: what breaks, how do we roll back, what's the blast radius, is there a backfill. Every one of those questions transfers.

The most valuable of those is the narrowing question. Additive clarification ("returns at most 50 rows") is usually safe. Narrowing language — adding a qualifier, a domain, a "use this only when" — changes tool selection, which is the behavior nobody reviews and everybody notices later.

🔍 What this looks like on a real diff

Say a data team runs a warehouse agent that answers ad-hoc questions over a metrics store. Someone notices the agent occasionally queries a huge raw events table when a rollup would do, and opens this PR:

 {
   "name": "query_events",
-  "description": "Run a read-only SQL query against the raw events table.",
+  "description": "Run a read-only SQL query against the raw events table. Expensive — prefer query_daily_rollup where possible.",
   "parameters": {
     "sql": { "type": "string", "description": "A SELECT statement." }
   }
 }

Read as prose, this is obviously good: it is true, it is helpful, it fixes the reported problem. Read as a migration, three questions surface immediately.

1. What does "expensive" do to the risk-averse case? The model now has a reason to avoid the tool. Some questions genuinely need raw events — per-session ordering, rare event types that never made it into the rollup. Does the agent now refuse those, or answer them from the rollup and get them subtly wrong? That second failure is much worse than the slow query you were trying to fix.
2. Is query_daily_rollup actually able to absorb the traffic you just redirected? A description edit is a routing change. You have pointed load at a tool without checking its schema covers the redirected questions.
3. What is the rollback signal? "Cost went down" will look like success even if answer quality fell with it. The metric that has to be watched is the pair — cost and the rate at which the agent answers raw-events questions correctly.

The fix is not to reject the PR. It is to require the author to attach the same evidence a migration PR carries: a before/after run over a fixed set of maybe thirty recorded questions, split into ones that should hit the rollup and ones that must hit raw events. If both buckets route correctly after the change, approve it. That fixture set costs an afternoon to build once and pays for every description change afterward.

⚠️ Where the analogy stops working

A migration is deterministic. Run it on a staging copy and you know exactly what happens in production.

A description change is statistical. Your thirty-question fixture can pass while the change still shifts behavior on the long tail you didn't record. So the migration ritual gets you the review, but not the confidence — you still need the production-side half:

• Ship description changes alone, never bundled with code, so a behavior shift has one candidate cause.
• Log the tool-selection distribution, not just errors. A change that moves query_events from 30% of calls to 4% should be visible on a chart the day it ships.
• Keep the previous description in the PR body verbatim, so reverting is copy-paste rather than archaeology.

The one question to add to your review template

Most of this collapses into a single line worth adding wherever your team writes PR checklists:

Does this diff change what the model is allowed to do, or what it is likely to do?

The first kind — new tool, new parameter, widened permission — already gets scrutiny. The second kind is the one that ships on two thumbs-up and gets blamed on the model a week later. Tool descriptions are almost always the second kind, and they deserve the migration treatment precisely because they never look like they need it.