Request, Worker, or Durable Execution? Hosting Models for Agents

Where an agent run lives while it executes is a decision that constrains everything downstream — how long runs can be, whether they survive a deploy, whether they can wait for a human. It's usually made implicitly by putting the agent in whatever the team already runs, and then fought against for a year.

Three models, with a clear rule for choosing.

In the request

The agent runs inside an HTTP request handler. The user's connection stays open; the response is the result.

Fits: short interactive runs, a few turns, under about thirty seconds.

Breaks at: anything longer. Request timeouts, load balancer limits, and browser behavior all conspire. A deploy mid-request kills the run with no record. There's no retry, no resumption, and no way to wait for anything.

The trap is that it works fine in development, where runs are short and nothing is deployed mid-request, and starts failing in production as the agent grows more capable and its runs grow longer.

Background worker

The request enqueues a job and returns a run ID. A worker pool executes; the client polls or subscribes for updates.

Fits: most agent workloads. Runs of seconds to minutes, retryable, survives client disconnection, and scales by adding workers.

Requires: a queue, a run-state store, a way to deliver progress and results, and a client that handles asynchrony.

Breaks at: waits longer than a worker should hold — hours for a human approval, or an overnight external job. Holding a worker for that is wasteful and fragile, and a deploy still interrupts anything in flight.

This is the right default for the large majority of agent products. ⚠️ The main thing teams get wrong here is not making runs resumable, so a deploy or a crash loses work that was nearly complete.

Durable execution

The run's state is checkpointed at every step, so execution can stop and resume anywhere — across process restarts, deploys, and arbitrary waits. Whether you use a workflow engine or build it, the property is the same: progress is persisted, not held in memory.

Fits: long-running runs, human-in-the-loop waits, multi-day processes, anything needing exactly-once side effects across restarts.

Costs: real complexity. Steps must be idempotent, state must be serializable and versioned, and debugging spans checkpoints rather than a single stack. There's also a schema-migration problem nobody warns you about — runs in flight when you deploy a change to the state shape.

The choosing rule

Work from the longest wait your agent can encounter:

• Under 30 seconds, always → in the request. Keep it simple.
• Seconds to minutes, no external waits → background worker. Default choice.
• Any wait for a human, or for an external job measured in hours → durable execution. Nothing else handles it without a workaround you'll regret.
• Side effects must be exactly-once across restarts → durable execution.

The mistake in both directions is real: durable execution for a five-second agent is heavy machinery around nothing, and a background worker for a workflow with an approval step produces a system that leaks half-finished runs.

✅ Design so the choice can change

Whichever you pick, a few habits keep migration cheap:

• Keep run state out of process memory. In a store, from the beginning. This is the single decision that makes every later migration possible.
• Make steps idempotent. Needed for durable execution, useful everywhere, and expensive to retrofit.
• Separate the loop from the transport. The agent loop shouldn't know whether it's in a request handler or a worker.
• Emit progress events from the start, even when nobody consumes them. Adding them later means touching everything.

Do these and moving from request to worker is a small change, and worker to durable is a contained one.

💡 The hybrid most products land on

Interactive requests run in a worker with a short budget, so the user gets an answer quickly. Anything that needs approval, hits a long external wait, or exceeds its budget is promoted to a durable run and continues asynchronously, notifying the user when it completes.

This gives fast interactive behavior for the common case without the ceiling that pure request-scoped or pure worker-scoped execution imposes — and it requires exactly the four habits above.

The takeaway

Pick by the longest wait the run can encounter, not by what's already deployed. Under thirty seconds, in the request; minutes without external waits, a worker; anything involving a human or a long external job, durable execution. Then keep state out of memory, make steps idempotent, and separate the loop from the transport — so when the answer changes, the migration is a change rather than a rewrite.