Your Agents Asked 40 Questions This Week. Which Ones Should Have Reached a Human?
Human attention is the one resource in an agent deployment that never shows up on the bill — so agents spend it freely. Here's a three-way routing rule, a threshold you can actually tune, and the two metrics that tell you when your agent is asking too much or too little.
A scheduling agent runs for a team of six. On Friday, the shared review queue holds forty messages from it: "Should I book the 30-minute or the 45-minute slot?", "The attendee list has two people named Sam — which one?", "Is Tuesday's offsite an all-day block or does it end at 3?". One of those forty was worth asking. The other thirty-nine were decisions the agent could have made, logged, and moved on from — and by burying them, it made the one that mattered arrive late.
The interrupt is the most under-priced resource in an agent deployment. Token cost gets a dashboard. Latency gets an alert. Human attention gets nothing, because it does not show up on the bill.
⚙️ Price the interrupt before you design the gate
Every question the agent raises costs three things, not one:
- The human's context switch. Reading the question, reconstructing enough state to answer it, answering, returning to what they were doing.
- The agent's parked run. The task is now blocked on a human's inbox latency, which is measured in hours, not seconds. Any state it was holding either persists (storage, staleness risk) or is thrown away (redo cost).
- The attention debt on the queue itself. Every low-value question makes the next reader skim harder. This is the expensive one, and it compounds.
Say a routine question costs a human ten minutes of real attention once the switch is counted. Forty of them is close to a working day, spent to resolve choices where either answer would have been fine. That is illustrative arithmetic, not a measured figure — but run it against your own queue depth and the number stops feeling abstract.
The question that should have been a logged decision
The test is not "am I confident?" Model confidence is a terrible gate: it is poorly calibrated, it drifts with prompt phrasing, and it tells you nothing about what happens if the guess is wrong.
The useful test has two parts:
- Is there a defensible default? Can the agent name a specific choice and a reason drawn from context — the team's usual meeting length, the Sam who is on the invite thread already?
- What does being wrong cost, and can it be undone? A 30-minute slot that should have been 45 is a two-click fix. An email sent to the wrong Sam is not.
If a default exists and the wrong answer is cheap and reversible, the correct behavior is not to ask. It is to decide, act, and write down what was assumed.
A three-way routing rule
Most systems offer the agent two options — ask, or silently proceed. The missing third option does most of the work.
| Route | When | What the human sees |
|---|---|---|
| Decide and log | Default exists; wrong answer is cheap and reversible | Nothing now; one line in a weekly digest |
| Ask | No defensible default, or the wrong answer is expensive or irreversible | A single question, with a proposed answer attached |
| Refuse and hand back | The task premise itself is wrong or the agent lacks authority | The task, returned, with what blocked it |
Expressed as a gate the agent evaluates before every ambiguous branch:
default = infer_default(ambiguity, context)
if default is None:
return ASK(question, proposed=None)
if not reversible(action) or blast_radius(action) > TEAM_THRESHOLD:
return ASK(question, proposed=default)
log_assumption(ambiguity, default, evidence)
return PROCEED(default)
TEAM_THRESHOLD is a real dial, and it belongs to the team, not to the prompt. Set it low while trust is being built; raise it as the assumption log proves boring.
💡 Make every question answerable in one word
An agent that asks "How long should this meeting be?" has handed the whole problem back. An agent that asks "I'm booking 45 minutes based on the last three reviews with this group — confirm or correct?" has done the work and kept only the judgment.
This turns a ten-minute interrupt into a fifteen-second one, and it makes the question self-documenting when it lands in the queue at 9pm and gets read at 8am. Require it in the output schema: a question without a proposed field is a malformed question.
Instrument the interrupt rate like any other production metric
Two numbers tell you whether the threshold is set correctly:
- Interrupt rate — questions raised per completed task. Trending up means the agent is hitting ambiguity it cannot resolve; that is usually a missing context source, not a reasoning failure. Chase the source, not the prompt.
- Override rate on logged assumptions — how often a human, reviewing the digest, says the default was wrong.
Read them together:
- Override rate near zero → the agent is being too cautious somewhere. Raise the threshold; convert some ask branches to decide and log.
- Override rate climbing → lower the threshold, and look at which assumptions get overturned. They cluster, and the cluster names the context the agent is missing.
⚠️ An interrupt rate of zero is not a win. It usually means the escalation path is broken, or the agent has learned that asking is punished and has started guessing on the irreversible branches too.
The digest is the real gate
Teams tend to obsess over the ask path and neglect the log. That is backwards. The ask path is self-correcting — a human sees the question. The decide-and-log path is where silent drift lives, and it only surfaces if someone actually reads the assumption stream.
Give it a slot: one person, once a week, skimming what the agent decided on its own. That review is cheap precisely because thirty-nine questions became thirty-nine lines instead of thirty-nine interrupts.
The goal was never an agent that never asks. It is an agent whose questions are rare enough that people still read them.