Designing for Resumability: Agents That Survive a Restart

A migration agent was forty minutes into a two-hour job when the process was restarted for an unrelated deploy. It began again from the top — re-reading files it had already converted, re-applying changes that were already applied, and producing a result that took longer to untangle than doing the whole thing by hand.

Long-running agents need resumability, and it isn't something you add afterwards. It's a property of how you structure the run.

What actually has to survive

The instinct is to serialize the message array and reload it. That's necessary and not sufficient — and on its own it produces the subtly-wrong resumption above.

Four things must be recoverable:

1. The work ledger — what was requested, what's completed, what remains. Owned by the loop, not inferred from the transcript.
2. Side effects already applied — which writes actually landed, so they aren't repeated.
3. Enough context to continue — not necessarily the full history; enough to make the next decision well.
4. The position — which item, which step, which attempt.

Reconstructing any of these by re-reading the message array is where resumption goes wrong. A transcript says what the agent said it did, which is not the same as what completed — especially if the crash landed between a tool call and its result.

Structure the run as items, not as a conversation

The pattern that makes everything else easy: model the work as an explicit list of items with states, held outside the message array.

run_id: r_8821
items:
  - {id: file_012, state: done,        result_ref: commit_a1}
  - {id: file_013, state: in_progress, attempts: 1, started: ...}
  - {id: file_014, state: pending}
  - {id: file_015, state: pending}

Resumption is now trivial and, more importantly, correct: anything done is skipped, anything in_progress is re-attempted or escalated, pending continues. The message array becomes a per-item working buffer you can discard rather than the system of record.

This also gives you progress reporting, per-item retry limits, and parallelism for free — all of which you'd otherwise build separately.

The in-progress problem

The interesting state is in_progress at crash time. Did the side effect land or not?

Three approaches, in order of preference:

Make the operation idempotent. Include a deterministic key derived from the item — file_013@run_r8821 — and have the receiving system deduplicate. Then re-attempting is always safe and you don't need to know what happened. This is the only approach that's fully correct under partial failure, and worth real effort to arrange.

Write intent before acting. Record "about to do X" durably, then act, then record completion. On resume, an intent with no completion means "check whether X happened." You still need a way to check, but you know exactly what to check.

Verify before repeating. On resume, read the target state and compare. Works when effects are observable; fails for things like sent email, where the effect is real and unobservable after the fact.

⚠️ Notice all three require the tools to cooperate. Resumability is largely a property of your tool layer, not your loop — which is why bolting it on later means rewriting tools.

Context on resume: rebuild, don't restore

The temptation is to reload the full message array and continue mid-conversation. For a long run this restores a bloated context, including turns that are now irrelevant.

Better: rebuild a fresh context from durable state — the original task, the constraints, the ledger summary ("42 of 60 items complete; 3 failed with reason X"), and the current item. This is usually far smaller than the array you saved, and often produces better subsequent behavior than the original run had, because the accumulated noise is gone.

💡 A pleasant side effect: if you can rebuild context from state, you can also compact mid-run using the same code path. Resumability and long-run compaction turn out to be the same mechanism.

What to write down, and when

• After every state transition, not at the end. Cheap; a run that fails writes the same records as one that succeeds.
• Include a schema version on the state. You will change the shape while runs are in flight.
• Record the model version and prompt version with the run. Resuming with different behavior mid-job is a real and confusing failure.
• ✅ Make resumption a tested path, not a hoped-for one. Kill a run halfway in CI and resume it. This test finds bugs immediately and continuously.

The takeaway

Resumability isn't checkpoint-and-reload. It's modelling the work as items with durable states, making side effects idempotent or verifiable, and rebuilding context from state rather than restoring a transcript. Do this and restarts become uneventful — and you get progress reporting, retry limits, and mid-run compaction as a bonus, because they all read from the same ledger.