Fan-Out, Fan-In: Parallelism Inside a Single Agent

An agent that must check twelve services, read eight documents, or process forty records will, left to itself, do them one at a time — one tool call per turn, each turn re-sending the whole context. Twelve sequential turns for work that has no sequential dependency at all.

Parallelism here doesn't require multiple agents. It requires the loop to recognize independent work and dispatch it together, and the design has a few non-obvious details.

The cheap version: parallel tool calls in one turn

Most tool-calling APIs let a model request several calls in a single turn. Executing them concurrently is a change in your dispatch layer, not in the model:

if len(reply.tool_calls) > 1:
    results = await gather(*[run(c) for c in reply.tool_calls])
else:
    results = [run(reply.tool_calls[0])]
messages.extend(results)

Twelve service checks become one turn with twelve concurrent calls. Wall-clock drops to the slowest call rather than the sum, and — the part that's easy to miss — you pay for the context once instead of twelve times.

For this to happen the model has to actually request them together, which is mostly a matter of not discouraging it. A prompt that says "check each service in turn" produces serial calls; describing the tool as batchable and the work as independent produces parallel ones.

⚠️ Concurrency needs a cap. Twelve is fine; four hundred against a rate-limited API is an incident. Put a semaphore in the dispatch layer, sized to the downstream limit, not to the number of calls requested.

The better version: a batch tool

If the work is uniform, one tool that takes a list beats N parallel calls:

check_services(names: string[]) -> [{name, status, latency_ms, last_deploy}]

Advantages over parallel dispatch: one result the model reasons over as a set, far fewer tokens than N separate result blocks, and the concurrency control lives in your code where it belongs. → When the calls are homogeneous, prefer the batch tool. Reserve parallel dispatch for genuinely heterogeneous work.

Fan-out to sub-runs, when each item is substantial

For work where each item needs several steps — read a document, extract, evaluate — a batch tool doesn't fit. Here each item gets its own sub-run:

results = await gather(*[
    run_subtask(item, tools=SUBSET, max_turns=6) for item in items
])

The design questions that matter:

What does each sub-run get? Not the parent's full context — that defeats the purpose. The item, the standing constraints, and whatever shared reference material it needs.

What does each return? Structured, small, uniform. This is the fan-in, and it's where the pattern usually goes wrong: if each of twenty sub-runs returns three paragraphs of prose, the parent now has sixty paragraphs and you've moved the context problem rather than solved it.

{ item_id, status: "ok"|"failed"|"needs_review",
  finding: string,          # one or two sentences, capped
  evidence_ref: string,     # pointer, not the evidence itself
  confidence: "high"|"low" }

What happens when one fails? Decide explicitly: fail the batch, continue with partials, or retry that item. Continuing with partials is usually right — but only if the final output states which items failed. A summary over eighteen of twenty items presented as complete is the silent-partial-result failure again.

The fan-in step deserves its own attention

Aggregating N structured results is the part people write last and think about least. Two things help:

Aggregate in code where you can. Counting, grouping, sorting, and filtering the results don't need a model. Compute the summary statistics deterministically and give the model the aggregate plus the outliers, rather than all N records to summarize.

Preserve item identity. Every finding in the final output should trace back to which item produced it. Without that, a wrong conclusion in the summary can't be traced to the sub-run that caused it.

✅ When to reach for this

• The work is a list, known before you start.
• Items don't depend on each other's results.
• Per-item work is more than one trivial call.
• The list is long enough that sequential turns hurt — usually somewhere above five.

❌ When not to

• Items inform each other. Finding X in item three should change how items four through twenty are read; parallel sub-runs can't do that.
• The list is short. Two items in parallel saves little and adds a failure mode.
• Order matters. Parallel completion order is arbitrary; if downstream cares, sort explicitly.

The takeaway

Independent work shouldn't cost sequential turns. Execute multi-call turns concurrently with a semaphore, prefer a batch tool for homogeneous work, and fan out to sub-runs when items need real multi-step handling. Then spend your attention on the fan-in: small uniform structured returns, aggregation in code, item identity preserved, and explicit handling of the items that failed.