Designing a Kill Switch That Works Mid-Run

Every agent deployment eventually needs someone to be able to stop it. Most kill switches are discovered to be inadequate during the incident they were built for, because stopping an agent is not the same as stopping a process, and nobody tested the difference.

Three things that all get called "stop"

Stop this run. One task, now. The user or an operator wants this particular agent to cease.

Stop all runs of this kind. A bad prompt shipped, a tool is returning garbage, a dependency is corrupt. Everything using this configuration must halt, including runs already in flight.

Stop new runs, drain the rest. The safe-shutdown case: no new work, let in-flight tasks complete. Appropriate for a deploy, not for an incident.

These need different mechanisms, and a system with only the first has nothing to reach for when the second is needed at 2am.

The check has to be in the loop

A kill switch implemented as process termination has a problem: an agent killed mid-tool-call leaves the side effect in an unknown state. Did the email send? Did the write commit? Nobody knows, and the incident now includes an audit.

The workable design puts a check inside the loop, between steps:

while True:
    if control.should_stop(run_id):        # cheap check, every iteration
        return halt_gracefully(run_id)     # record state, release locks, report
    reply = model(messages, tools=specs)
    if not reply.tool_calls: return reply.text
    for call in reply.tool_calls:
        if control.should_stop(run_id):    # again, before each side effect
            return halt_gracefully(run_id)
        messages.append(run(call))

Two check points matter: before each model call, and — critically — before each tool call. The second is what gives you a stop that never lands in the middle of a side effect.

⚠️ Process termination remains the last resort, and it should be understood as the option that leaves you with an unknown state, not as the primary mechanism.

What "graceful halt" must do

Stopping isn't just returning. A halted run should:

• Record its state at the point of stopping — what was done, what was in flight, what remained. Otherwise you can't resume and can't reconstruct.
• Release anything held — locks, claimed queue items, reserved resources. Runs stopped mid-flight are the classic source of stuck queues afterwards.
• Report the halt distinctly from success and from failure. A stopped run is its own outcome; collapsing it into "failed" corrupts your metrics on the day you most need them.
• Leave a resumable record, so the same run can continue later if the stop turns out to have been precautionary.

Make the switch reachable

A kill switch that requires a deploy is not a kill switch. Requirements, in rough order of how often they're missed:

Out-of-band. It must work when the system is unhealthy. A flag stored in the same database the agent is hammering, behind the same service that's failing, will not be available when needed. A separate, boring store — or a file, or a config service with a different failure domain — is worth the inelegance.

Fast to propagate. Seconds, not a cache TTL of minutes. In-flight runs checking a stale value is precisely the failure this exists to prevent.

Granular. By run ID, by agent type, by tenant, and globally. An incident affecting one customer shouldn't require stopping everyone, and an incident affecting one agent type shouldn't require stopping the platform.

Operable by whoever is on call. A documented command in the runbook, not knowledge that lives with the author. ✅ And tested in a drill, because a switch nobody has ever pulled is a hypothesis.

Related controls worth having alongside

A hard stop is the blunt instrument. Two softer controls prevent most situations from getting there:

• Per-run budgets — turns, tokens, wall-clock, and calls to any destructive tool. A run that exceeds its budget halts itself, which handles the runaway case without human involvement.
• A circuit breaker on tools. If a tool fails or returns anomalous results above a threshold across runs, stop offering it. This contains a bad dependency without stopping the agent entirely, and it reacts faster than a person can.

🔍 The drill

Once, deliberately: start a long run, stop it mid-flight, and check the aftermath.

• Did it stop within seconds?
• Was there a side effect in an unknown state?
• Were locks and claimed items released?
• Can you tell from the logs exactly where it stopped and what remained?
• Could you resume it?

Every gap this exposes is one you'd otherwise find during an incident, with an audience.

The takeaway

Stopping an agent means stopping it between actions, not during one. Put the check inside the loop before every model and tool call, halt gracefully with recorded state and released resources, keep the switch out-of-band and granular, and rehearse it once so it's known to work. And add per-run budgets and tool circuit breakers, so the switch is for the situations nothing else caught rather than for routine runaways.