Your Tool Returned 4,000 Rows and the Agent Needed Three: Truncate, Paginate, or Summarize?

A data agent gets a simple question: which vendors billed us more than usual last month? It calls query_invoices, the tool does exactly what it was built to do, and 4,000 rows of JSON come back. The agent needed three of them. The other 3,997 are now lodged in the conversation permanently, and every turn after this one pays to re-read them.

This is not really a context-window problem. The window is big enough. The problem is that the tool was designed for a caller that can loop, filter, and discard — and the caller it actually got reads everything once and never lets go.

🔌 The shape mismatch nobody designs around

A REST endpoint that returns a full result set is a fine endpoint. A Python client fetches it, runs a list comprehension, drops 3,997 rows, and moves on having spent nothing. That is the contract every API in your stack was written against.

An agent has no discard step. Whatever the tool returns enters the transcript verbatim and stays. Three things follow:

• Cost is multiplied by turn count, not paid once. Say the dump is 60,000 tokens and the agent takes eight more turns. That is closer to half a million tokens of input, not sixty thousand.
• The signal gets buried. The one anomalous vendor sits on row 2,891, surrounded by thousands of unremarkable rows. Attention over a long uniform span is exactly the condition where a single outlier goes unnoticed.
• The next call gets worse, not better. Having swallowed everything, the model tends to reason over the dump rather than issue the narrower second query it should have.

So the response has to shrink. The interesting question is how, because the three available moves are not interchangeable — each one destroys a different thing.

Truncate: cheap, and quietly wrong

Truncation is the default because it is one line of code: take the first N rows, append a note, return.

rows = db.query(sql)
if len(rows) > 50:
    return {
        "rows": rows[:50],
        "returned": 50,
        "total_matched": len(rows),
        "truncated": True,
        "hint": "Add filters or a sort order to narrow this."
    }

Truncation is correct when the result set is ordered by relevance and the tail genuinely doesn't matter — top search hits, most recent log lines, nearest matches. It is actively dangerous when the result set is unordered or the task is an aggregate. "Which vendors overbilled" cannot be answered from the first 50 rows of an arbitrary ordering, and the agent will happily answer anyway.

The failure mode is silent wrongness, so the fix is to make silence impossible:

• ✅ Always return total_matched alongside what you sent. The gap between 50 and 4,000 is the agent's cue to narrow.
• ✅ Say what the ordering is. "First 50 by amount descending" is usable; "first 50" is a trap.
• ❌ Never truncate the response body of an aggregate query. If the question is a sum, compute the sum server-side and return that instead.

Paginate: correct, and expensive in a way that hides

Pagination preserves everything. That is its appeal and its cost. Each page is a full round trip — model call, tool call, response back into context — and the pages accumulate anyway. Ten pages of 400 rows costs strictly more than one dump of 4,000, because you have paid ten times for the reasoning turns in between.

Pagination earns its keep in one situation: when the agent can stop early. A cursor-based tool where the agent scans page one, finds the record it needed, and abandons the cursor is a real win. A tool where the task inherently requires all pages is pagination as theatre — you have converted one large cost into a larger one spread across more turns, plus ten chances for the loop to go wrong.

💡 The diagnostic: before adding pagination, ask what makes the agent stop. If you cannot name a stopping condition the agent can evaluate from the current page, pagination is the wrong tool.

Summarize: the one that actually fits agents

The third option is to stop returning rows at all and return the shape of the answer instead. The tool does the aggregation, and hands back something small and already useful.

For the vendor question, the shaped response is roughly:

{
  "period": "2026-08",
  "vendors_evaluated": 312,
  "flagged": [
    {"vendor": "Northwind Freight", "billed": 48200, "trailing_3mo_avg": 19400, "ratio": 2.48},
    {"vendor": "Halcyon Print", "billed": 9100, "trailing_3mo_avg": 4050, "ratio": 2.25},
    {"vendor": "Ridge Data Co", "billed": 61000, "trailing_3mo_avg": 31500, "ratio": 1.94}
  ],
  "threshold": "ratio > 1.9",
  "detail_handle": "q_8823f1"
}

Under a hundred tokens, and it answers the question. The detail_handle matters as much as the numbers: it is a reference the agent can pass back to a fetch_detail tool if it needs the underlying rows for one vendor. The full result set still exists — it just lives outside the transcript until something asks for it.

This is the pattern most tool authors skip, because it requires deciding what the tool is for. A generic query_invoices cannot summarize; a find_billing_anomalies can. Narrow, opinionated tools shrink their own output almost for free.

Picking, in one pass

Run the tool's realistic result set through these questions in order:

1. Can the answer be computed server-side? → Summarize. Return the computed answer plus a handle.
2. Is the result ordered by relevance, with a tail that genuinely doesn't matter? → Truncate, with total_matched and the sort order stated.
3. Can the agent decide to stop after any given page? → Paginate with a cursor.
4. None of the above? → The tool is too generic. Split it into task-shaped tools until one of the first three applies.

That last branch is the one worth sitting with. A tool that cannot answer any of the first three questions is usually a database query wearing a function-calling schema. It works in testing, where the fixture has twelve rows, and falls over in production, where the same call returns four thousand.

The rule underneath

Tool responses are prompt content. Every field you return is something the model must read, on this turn and every turn after. An API contract optimizes for completeness because the caller can afford to ignore things; an agent tool contract has to optimize for relevance per token, because the caller cannot ignore anything.

The practical test before shipping any agent tool: run it against production-scale data, count the tokens in the response, and ask whether a careful human reader given only that text could act on it. If they would skim past most of it, so will the model — but you will have paid for every line.