The Analytics Agent's SQL Was Correct. It Answered a Different Question. Make It Print Its Definitions Before Its Number.

A product manager asks the analytics agent: "What was customer churn last quarter?" Ninety seconds later there is a tidy number, a chart, and the SQL that produced it. The SQL is valid, the joins are right, and a reviewer who checks it finds nothing wrong. The number is still wrong for the purpose it was used for, because the agent decided that "customer" included free-trial accounts and that "churn" meant any account with no login in 30 days.

Nobody made a mistake anyone could point at. The agent picked a reasonable reading of an ambiguous question and nobody saw the reading, only the result. This is the most common way data agents fail in practice, and it is not a SQL problem.

Why reviewing the query does not catch it

Teams that deploy a text-to-SQL or analytics agent usually add one safeguard: show the generated query so a human can check it. That catches syntax errors, bad joins and missing filters. It does not catch a wrong interpretation, for three reasons.

• The interpretation is spread across the query. "Customer" is a WHERE plan != 'free' clause, or its absence. "Last quarter" is a date expression. "Churn" is a subquery. Nobody reconstructs the business meaning from forty lines of SQL.
• Absence is invisible. The trial-account mistake is a filter that is not there. Reviewers are good at spotting wrong lines and bad at noticing missing ones.
• The number anchors the reader. Once a plausible figure is on screen, people read the SQL looking for confirmation, not for alternatives.

The fix is to move the interpretation out of the query and into a short, plain-language block the agent must produce first, and that travels with the answer.

🔍 The definitions block

Before the agent writes any SQL, it fills in a fixed structure for the question. Every field must be answered, and every answer must say where it came from.

QUESTION: What was customer churn last quarter?

METRIC:      churn rate = accounts cancelled in period / accounts active at period start
  source:    metrics catalogue, "churn_rate_v2"
POPULATION:  paying accounts only (plan != 'free', excludes internal test orgs)
  source:    metrics catalogue definition of "customer"
PERIOD:      2026-04-01 to 2026-06-30, calendar quarter, UTC
  source:    ASSUMED - "last quarter" read as the last complete calendar quarter
GRAIN:       one row per account (not per seat, not per workspace)
  source:    metrics catalogue
EXCLUSIONS:  accounts migrated between plans count as retained
  source:    ASSUMED - no catalogue rule found

Two things make this work.

Every field has a source, and "ASSUMED" is a legal value. The agent is not asked to be right about everything. It is asked to say which parts came from an agreed definition and which parts it guessed. A block with two ASSUMED lines tells the reader exactly where to look; a block with none tells them it is safe to skim.

The fields are fixed. Metric, population, period, grain and exclusions cover most of the ways an analytics question goes ambiguous. Letting the agent choose which dimensions to mention means it will mention the ones it was confident about and skip the rest.

Building it into the loop

The definitions block is a step in the agent's plan, not a paragraph in the final answer. A minimal version looks like this:

def answer_metric_question(question):
    defs = agent.fill_definitions(
        question,
        catalogue=metrics_catalogue.search(question),  # retrieved, not remembered
        required_fields=["metric", "population", "period", "grain", "exclusions"],
    )

    assumed = [f for f in defs.fields if f.source == "ASSUMED"]
    if any(f.name in {"metric", "population"} for f in assumed):
        return ask_user_to_confirm(defs)   # the expensive guesses get a checkpoint

    sql = agent.write_sql(defs)            # SQL is written from defs, not from the question
    result = warehouse.run(sql)
    return Answer(value=result, definitions=defs, sql=sql)

Three design choices matter here.

1. Definitions come from retrieval, not the model's memory. If your company has a metrics catalogue, a dbt semantic layer or even a wiki page of agreed definitions, the agent looks the term up. A model's general idea of "active user" is a guess dressed as knowledge.
2. The SQL is generated from the block, not from the original question. This keeps the query and the stated interpretation in sync. If the SQL writer sees the raw question, it can quietly re-interpret it and the block becomes decoration.
3. Only some assumptions stop the run. Guessing the time zone rarely changes a decision. Guessing who counts as a customer often does. Gate on the fields where a wrong guess changes the story, and let the rest pass with a visible flag, or the confirmation prompts become noise people click through.

✅ What the answer looks like now

The reply the product manager receives leads with the definitions, then the number:

Churn last quarter (Apr 1 to Jun 30, paying accounts only, per the catalogue's churn_rate_v2): [value]. Assumed: plan migrations count as retained. Change this?

That second line is the whole point. The PM who knows that finance treats downgrades as partial churn can correct it in one reply, before the figure is copied anywhere. Under the old flow, that correction happens weeks later, if ever, when someone else's report disagrees.

Where this goes wrong

• ❌ Letting the block get long. If it runs to twenty lines, nobody reads it. Five fields, one line each, sources in a few words.
• ❌ Writing definitions after the query. An agent asked to explain its SQL will produce a tidy account of what it already did. That is a summary, not a commitment, and it inherits every wrong assumption.
• ❌ Treating the block as a one-off. When a user corrects an ASSUMED field, that correction is a candidate for the catalogue. If nobody feeds it back, the agent will guess the same way next week.
• ⚠️ Hiding it in the UI. Collapsing the definitions under a "details" toggle puts you back where you started. They should sit directly above the number.

The takeaway

A data agent's most important output is not the number. It is its reading of the question, and that reading is usually invisible. Make the agent commit to it in plain language, say where each part came from, and put it in front of the person asking before the number can travel. Reviewing the SQL finds bugs. Reviewing the definitions finds the wrong question, which is the error that ends up in the board deck.