The First Hour of an Agent Incident

A customer forwards a screenshot: your support agent told them refunds are available for 60 days. The actual policy is 30. Nothing crashed, no error rate moved, no alert fired, and the agent has been answering refund questions all week.

Ordinary incident response assumes a stack trace and a deploy log. Here you have neither. The first hour needs a different order of operations, because the expensive question is not what broke — it is when did it start. Every minute spent guessing widens the set of customers you have to go back and correct.

Why these incidents start sideways

Three properties separate an agent incident from a service incident:

• Nothing crashed. A wrong answer is a successful request. Latency is fine, the status code is 200, the trace is complete. The incident enters through a human: a customer, a reviewer, someone reading a sampled conversation.
• The blast radius is historical. A broken endpoint stops hurting the moment it is fixed. A wrong policy answer sits in inboxes, and some of those conversations already ended in an action that now has to be unwound.
• The cause may not be a deploy. Which is why the timeline question is hard, and why it comes first.

Minutes 0–5: narrow, don't stop

The instinct is a kill switch. Resist it when you can. A full stop is an outage you chose to have, and it turns a quality problem into an availability problem in front of the same customers.

Identify the class of bad output, then reduce scope to exactly that class:

• Route refund and policy conversations to the human queue, leaving order tracking and address changes running.
• Or disable the one tool implicated — if the agent is quoting a policy lookup that returns stale text, cutting that tool makes it say "let me get a colleague" instead of confabulating.
• Or drop to a restricted mode where the agent drafts and a human sends.

Full stop is correct when you cannot yet describe the class, or when the agent takes irreversible actions and you cannot rule those out. Otherwise, scope reduction buys the same safety at a fraction of the cost.

⚠️ Whatever you flip, write down the exact timestamp. You will need it in twenty minutes to separate "the fix worked" from "the traffic pattern changed."

Minutes 5–20: find when it started

This is the phase people skip, and it is the one that determines how much cleanup you owe. Four things change underneath an agent, and only one of them looks like a deploy:

1. The prompt — a system prompt edit, often shipped from a console rather than a repo.
2. The tool surface — a tool description reworded, a parameter renamed, an API's response shape changed under a tool that still returns 200.
3. The model — a version pin moved, a provider rolled an alias forward, a fallback route started serving a different model during a capacity event.
4. The retrieved corpus — a document reindexed, a policy page edited, a stale copy re-crawled and now outranking the current one.

Build a single timeline with all four on it. Then find the earliest bad output you can confirm and read what changed just before it.

🔍 The high-yield query is content-based, not error-based: search past runs for the wrong claim itself. In the refund case, look for conversations whose final message contains a number-plus-day pattern in a refund context, bucketed by hour. The point where the rate jumps from zero is your start time, and it is usually sharper than any change log.

If you cannot run that query, that is the top finding of the postmortem. Everything else in this hour depends on it.

Then bound the damage: how many conversations in that window, how many customers, and how many ended in an action you cannot silently reverse — a refund issued, an account closed, a shipment cancelled.

Minutes 20–40: reproduce before you touch anything

Take one confirmed-bad conversation and replay it. Replay means the same input and the same tool responses that run actually received, not fresh calls — otherwise the corpus you are trying to indict has already changed under you.

replay(run_id) →
  same system prompt version
  same tool definitions
  recorded tool responses, not live calls
  same model version

Two outcomes, and they lead to different places:

• It reproduces. You now have a test case and something to bisect. Swap one variable at a time — old prompt with new model, new prompt with old corpus — and the responsible change usually falls out in three or four runs.
• It doesn't reproduce. Stop trying to make it happen on a fresh run and go read the raw message array of the actual bad run instead. What you find is often a retrieved chunk that no longer exists, or a tool response nobody expected: an empty result the agent filled in from memory, an error string it read as data.

Minutes 40–60: roll back or fix forward

Rollback is cheap here in a way it is not for stateful services. Prompt versions, tool definitions and model pins are all just versions; reverting them is a config change, and the corpus can be pointed at the previous index snapshot if you keep one.

Fix forward only when you can state the mechanism in one sentence. "The reindex pulled in an archived policy page that ranks above the current one" earns a fix forward. "The prompt could be clearer about policy" does not.

❌ The specific trap: editing the system prompt as the fix, without a reproduction. Prompt edits feel productive, ship in seconds, and are unfalsifiable at that moment — you cannot tell whether the incident stopped because of the edit or because the affected traffic moved on. An hour later you have a longer prompt, an unknown cause, and no way to write the postmortem.

If you must ship a prompt change before you understand the cause, label it in the channel as mitigation, not fix, and keep the incident open.

✅ What makes this hour fast, prepared in advance

None of this can be assembled during the incident:

• Every run logs prompt version, tool-definition version, model version and index version.
• Past runs are queryable by output content, not just by id and timestamp.
• Replay exists and uses recorded tool responses.
• A scope-reduction switch exists separately from the kill switch, at the granularity of intent or tool.
• A written list of which agent actions are irreversible, so "how bad is this" is a lookup rather than a debate.

The postmortem question that actually matters

For a service outage, the question is why it broke. For an agent incident, the better question is why a customer found it before you did. The detection gap is almost always larger than the defect, and it is the only part of this that generalizes to the next incident — which will be a different wrong answer, arriving the same way.