Which Agent Alerts Should Wake Someone Up?

Agent alerting is usually inherited from service alerting: error rate, latency, uptime. Those three miss most of what goes wrong with an agent, and page for several things that don't matter.

An agent can be entirely healthy by service metrics — no errors, normal latency, every request answered — while doing the wrong work at ten times the usual cost. It can also throw errors continuously as a normal operating mode, because a tool failing and being routed around is a handled path, not an incident.

The sorting rule that works: page only when a human has an action, right now, that no machine is going to take.

Wake someone up

A safety invariant fired in production. A forbidden tool was called, an approval gate was bypassed, a tenant boundary check failed. These should be impossible; if one happens, the assumption behind your whole permission model is wrong and the blast radius is growing while nobody looks.

Spend rate exceeds a multiple of baseline. ⚠️ This is the highest-value single alert an agent deployment can have, and the one most often absent. Agents can burn money silently and fast — a retry loop, a runaway multi-agent tree, a prompt change that doubled turns. A short-window rate alert (spend in the last fifteen minutes against the trailing average) catches it while the number is still small.

Blocked-egress attempts spiking. Near-zero is normal. A sustained rate means either an injection is working or someone is probing, and both need a person.

Duplicate side effects detected. If your idempotency reconciliation finds the same effect applied twice, something is replaying real actions. Money, mail, and tickets don't wait for morning.

Total failure of a critical path, where the circuit breaker has tripped and no degraded mode exists. The distinction matters — see below.

File a ticket for the morning

Real signals, none of which a person can act on faster at 3am than at 9am:

• Turns per completed task, p95, up beyond a threshold. The earliest indicator that something regressed, and the fix is a code or prompt change.
• Escalation rate moved sharply in either direction. Up means the agent lost confidence; down means it stopped asking when it should.
• Cache hit rate collapsed. Almost always a context-ordering change, and it costs money continuously until fixed.
• One tool's error rate above its own baseline. A degrading dependency — the circuit breaker is handling it, and someone should find out why.
• Completion rate down without an accompanying error spike. The quiet quality regression.

Dashboard only — never alert

• Individual run failures. Agents fail runs; that's what escalation and retry are for.
• Individual escalations. The mechanism working.
• Latency of any single call.
• Absolute token totals, which rise with usage and mean nothing on their own.

Alerting on any of these produces noise that trains people to ignore the channel, which is how the invariant page gets missed later.

💡 The circuit-breaker distinction

The test for whether something pages: has automation already contained it?

A dependency failing, with a breaker that tripped and a degraded path serving traffic, is a ticket. The system did what it was designed to do; a human has no better move at 3am.

The same dependency failing with no breaker and no degraded path is a page, because containment requires a person.

→ Which means every alert you promote to a page is a candidate for automation instead. "This wakes someone up" and "we haven't built the automatic response yet" are frequently the same statement, and the second framing is more useful.

What the alert has to contain

An agent page that says "error rate high" wastes the first ten minutes:

AGENT ALERT — spend rate 6.2× baseline
  feature:        invoice-extraction
  since:          21:14Z (27 min)
  driver:         turns/task p50 4 → 19
  correlates:     prompt v22 deployed 21:09Z
  example runs:   r_88213, r_88219, r_88240
  rollback:       agentctl prompt set invoice-extraction v21

What changed, when, the likely driver, example runs to read, and the command to undo it. The rollback line matters most: whoever is paged needs authority and a one-step action, or they're a relay to whoever has it.

🔍 The failures no alert catches

Some of the worst agent failures are invisible to any threshold — an agent confidently answering wrongly, filing things that needed a reply, resolving ambiguity in the wrong direction. Every metric looks fine.

Those need a weekly sampled review: a dozen completed runs read by someone who knows what right looks like, plus a sample of whatever the agent removed, closed, or declined. It's the only mechanism that catches quiet wrongness, and it doesn't scale — which is exactly why it needs to be scheduled rather than intended.

The takeaway

Page for violated invariants, runaway spend, egress attempts, duplicated effects, and uncontained failure. Ticket the metric regressions — turns, escalation rate, cache hits, per-tool errors. Dashboard the rest. Put the rollback command in the alert itself, and treat every page as a question about what should have been automated. Then schedule the weekly sampled read, because the failures that matter most produce no signal at all.