Draining a Data Backlog: Should the Agent Batch Rows or Take Them One at a Time?

A nightly export drops 12,000 unlabeled rows into a queue — raw support messages that each need a product area, a severity, and a routing tag. An agent with a classifier prompt and a database writer can drain it by morning. The choice that decides whether the job costs a few dollars or a few hundred, and whether a bad night corrupts 12,000 rows or 40, is unglamorous: does the agent handle one row per turn, or fifty?

Most teams pick one shape by instinct and never revisit it. The instinct is usually wrong in one direction or the other, and the tell is which number you optimized without measuring.

🔁 Two shapes of the same loop

One row per turn. The agent sees a single record, may call a lookup tool, decides, writes one row.

for row in backlog:
    verdict = agent.run(system=TAXONOMY, user=render(row))
    db.upsert(row.id, verdict)

Fifty rows per turn. The agent sees a chunk, emits a list of verdicts, and one bulk write lands them.

for chunk in chunks(backlog, size=50):
    verdicts = agent.run(system=TAXONOMY, user=render_many(chunk))
    db.bulk_upsert(verdicts)   # each verdict carries its own row_id

Same model, same taxonomy, same output schema. The difference is entirely in what gets repeated.

💰 The cost lives in the part you repeat

Every turn re-sends the system prompt, the tool schemas, and the label taxonomy. Suppose that fixed preamble is roughly 2,000 tokens and a row is roughly 150.

• One row per turn: ~2,150 input tokens per row.
• Fifty rows per turn: ~2,000 + 7,500 = ~9,500 for the chunk, or ~190 per row.

The variable part barely moved. The fixed part got amortized across fifty rows instead of one. That single ratio — preamble size divided by row size — is the whole economic argument for batching, and it takes five minutes to compute against your actual prompt before you write any code.

The ratio also tells you when batching is pointless. If each item is a 4,000-token document and the preamble is 400 tokens, you were already spending 90% of every call on content. Chunking buys you a rounding error and costs you everything in the next section.

Latency follows the same shape but for a different reason: per-turn round-trip overhead, queueing, and retry backoff are paid once per turn, not once per row. Two hundred and forty chunked turns finish a lot sooner than 12,000 sequential ones — though a single-row loop parallelizes across workers just fine, which is the escape hatch people forget when they reach for batching purely for speed.

⚠️ What batching quietly breaks

Alignment. The model returns 48 verdicts for 50 rows, or silently shifts one position after a malformed record. Positional matching turns that into 50 confidently wrong labels. Fix: every verdict object carries the row id back, and the writer rejects any verdict whose id wasn't in the chunk it sent. Never zip by index.

Retry amplification. One poisonous row fails the chunk. Retrying the chunk re-pays for 49 rows that were already correct. Do that a few times a night and the amortization you bought back gets spent on re-work. The mitigation is a fallback path: a chunk that fails twice gets split, and its rows drop into the single-row lane.

Blast radius. In a single-row loop, a drifted instruction or a bad taxonomy edit shows up in the first few rows and a sampling check catches it. A batched loop writes 50 wrong rows before anything can look at one.

Attention dilution. Long lists invite terser, lazier answers near the end — a real effect worth measuring rather than assuming. The measurement is cheap: take 50 rows, run them as 50 singles and as one chunk of 50, and compute the disagreement rate. That number, on your data with your prompt, settles the argument better than any general claim, including this one. Repeat it at chunk sizes of 10, 25, and 50 and you'll find the knee.

🎯 The hybrid is usually the right answer

Routing by difficulty beats a uniform policy. A cheap pre-filter — rules, an embedding-distance check against known clusters, or a small model — sorts the backlog into two lanes:

• Boring lane → batched. Short, typical, high-confidence rows. Chunked at whatever size your disagreement test justified. No per-row tool calls.
• Interesting lane → single. Ambiguous rows, unusually long text, unseen vocabulary, and anything touching money, a customer commitment, or an escalation. One turn each, tool lookups allowed, one trace per row.

A canary helps too: run the first 50 rows of every night singly, compare the label distribution to yesterday's, and only unlock batching if it looks sane. It costs one chunk's worth of budget to avoid a 12,000-row cleanup.

✅ The checklist

Batch when:

• The preamble dwarfs the row.
• Rows are independent — no row's answer changes another's.
• The output is small and schema-shaped, one object per row id.
• The write is idempotent per row id, so a partial re-run is harmless.
• A wrong label is cheap to correct later.

Go one at a time when:

• Each row needs its own tool calls whose results steer the next decision.
• Rows are ordered or interdependent.
• The action is irreversible — a refund, an email, a ticket closure.
• You need a per-row trace for audit or debugging.

🔍 Watch cost per correct row

Cost per row is the headline number and the misleading one. The dashboard that actually tells you whether batching is working tracks four things: disagreement rate against a single-row control on a held-out sample, chunk retry rate, rows reprocessed per row completed, and the downstream correction rate — how often a human fixes a label the agent wrote.

A batched pipeline running at a fifth of the token cost with three times the correction rate is not a saving; it moved the spend from your API bill to somebody's afternoon.

Batching compresses fixed cost. It does not compress the work, and it does not make the model more careful. Treat chunk size as a dial tuned against your own disagreement curve, keep a single-row lane open for the rows that deserve it, and let the boring 90% go through in bundles.