Where Should the Agent Loop Live? Framework, Job Queue, or Workflow Engine

An alert-investigation agent takes about four minutes on a good run: read the alert, pull recent metrics, check the deploy log, look at two dashboards, write a summary. On a bad run — a noisy alert, a slow query, a service that won't respond — it takes forty.

The first version ran the loop inside a web request. It worked until the first afternoon deploy, which killed nine investigations mid-flight, three of which had already restarted a service.

Where the loop physically runs is usually decided last, after the prompt and the tools. It should be decided by one question: how long is a run relative to how often the process restarts, and what does a crash at step six leave behind?

Option 1: the framework's in-process runner

The loop runs in the process that started it. Most agent frameworks ship this and it is the right default for building.

What it gives you: the shortest path to a working agent, a stack trace that contains the whole run, and streaming that costs nothing because the tokens are already in the process that holds the connection.

What it costs: every deploy kills every run in flight. There is no external record of what was mid-flight, so recovery is a person reading logs. And the unit of scale is a whole process — thirty concurrent forty-minute investigations means thirty of them resident at once, mostly waiting on network.

✅ Correct when a run is short enough that losing one is merely annoying, or while the shape of the loop is still changing daily.

Option 2: one step per job on the queue you already have

Each iteration of the loop becomes a job. The state — the message array, the step count, the budget spent — lives in a row. A job runs one step, writes the new state, enqueues the next job.

job: agent_step(run_id)
  state = load(run_id)
  msg   = model(state.messages)
  if msg.tool_calls:
      state.messages += run_tools(msg)
      save(state); enqueue(agent_step, run_id)
  else:
      save(state, done=True)

The appeal is that everything the queue already does becomes free: retries with backoff, concurrency caps, backpressure when the provider rate-limits, a dead letter queue, and dashboards the ops team already knows how to read. A deploy drains workers and the run picks up on the next one.

Two real costs. The transcript round-trips through the database on every step, and agent transcripts get large — write amplification is genuine and worth measuring before it surprises you. And the control flow is now scattered across job handlers, so reading "what does this agent do" means reconstructing it from enqueues rather than reading a loop.

✅ Correct when the queue already exists, runs last minutes, and someone else already operates the infrastructure.

Option 3: a durable workflow engine

Durable execution frameworks resume a crashed run at the step it reached. That is exactly what a forty-minute agent run wants, and it is why they keep coming up in these conversations.

⚠️ The constraint that surfaces around day three: these engines resume by replaying your code from the top and skipping the steps whose results were already recorded. Replay assumes the code is deterministic. A model call is the canonical nondeterministic effect, and so is every tool call that touches the outside world.

So each one has to be wrapped as a recorded activity, and the workflow body can only branch on values that were recorded. Get this wrong and a resumed run silently diverges from the run that crashed.

result = await record_activity(call_model, messages)   # recorded, replayed from log
if result.tool_calls:                                   # branching on a recorded value: fine
    obs = await record_activity(run_tool, result.tool_calls[0])

The practical consequence is stylistic and it bites: the loop becomes a sequence of explicitly recorded steps rather than ordinary code, and changing how the loop is shaped is a deploy plus a decision about what happens to runs mid-flight under the old version. Prompt-level experimentation gets slower.

The comparison

🔍 Two questions that pick for you

Is someone watching the run happen? A person waiting on an answer wants tokens now. In-process gives that for free; the other two need an explicit channel for partial output, which is real work.

Does a lost run leave the world half-changed? This one matters more. An investigation agent that only reads is safe to lose — rerun it. An agent that has already scaled a deployment, silenced an alert, or opened a ticket, and then dies, has left inconsistent state that a human now has to reconcile without knowing what the agent intended to do next.

Any agent that takes actions with external effects should be able to resume or to roll back. Neither is available in-process.

The migration that usually happens, in order

1. Start in-process. The loop's shape is still changing and iteration speed is worth more than durability.
2. Move to the queue when deploys start killing runs that matter, or when concurrency starts costing memory.
3. Reach for the workflow engine when partial completion becomes dangerous rather than merely wasteful.

❌ Skipping to step three early is the common mistake. Paying the determinism constraint while still rewriting the loop every day means every experiment carries a versioning question, and the thing being protected — a stable multi-step process — does not exist yet.

Pick by what a crash at step six costs. If the answer is "rerun it," keep the loop simple. If the answer is "someone has to work out what it already did," the loop needs to outlive the process.