The Inbox Agent That Filed Everything Correctly and Lost Two Questions

An inbox agent triaged a week of email. Every decision it made was defensible against the rules it had been given. Two questions that needed answers ended up in the archive, and nobody noticed until the people who'd asked followed up.

Both threads ended with a message like "Thanks, that's helpful — no rush!" The agent read that as a conversation wrapping up. Three messages earlier, buried under the pleasantries, someone had asked something that was never answered.

The mistake: classifying the message, not the thread

The agent's triage step looked at one email and decided what to do with it. That's the natural way to build it, because that's the shape of the data — the API returns messages, and a message is what a classifier gets handed.

But "does this need a reply?" is not a property of a message. It's a property of a conversation. The last message in a thread tells you how the conversation sounds, not where it stands. A polite closing sentence sitting on top of an unanswered question looks, in isolation, exactly like a polite closing sentence.

What changed when the thread became the unit

Instead of classifying each message, the agent fetched the whole thread and answered a different question: what state is this conversation in?

classify_thread(messages) -> {
  state: "awaiting_me" | "awaiting_them" | "closed" | "fyi",
  reason: string,
  open_request: string | null    # the specific thing not yet answered
}

The open_request field is the one that fixed it. Asking for the specific unanswered request forces the model to look back through the thread rather than react to the most recent tone. If it can name a question nobody has answered, the state is awaiting_me regardless of how cheerful the last message was.

→ Before: "Last message is a thank-you → closed → archive." → After: "Message 3 asks which vendor we picked. No later message answers it. State: awaiting_me."

Three more things that made it usable

Never archive — stage. For the first few weeks, nothing was archived outright. Threads the agent judged closed went into a "ready to archive" label, reviewed in one pass at the end of the day. Reversible by default is what makes an agent like this adoptable at all; the version that acts irreversibly on day one has to be perfect immediately, and it won't be.

Explain every decision in one line. Each triaged thread carried its reason. Scanning fifty one-line reasons takes a minute and makes wrong calls obvious. A label with no reasoning has to be trusted or re-derived, and neither is fast.

Treat "unsure" as a real answer. Threads the agent couldn't confidently classify were left alone, in the inbox, untouched. Doing nothing is a legitimate and underrated output — an agent that handles seventy percent of threads and visibly skips the rest is more useful than one that handles everything with occasional silent mistakes.

🔍 Measuring the thing that actually matters

The obvious metric — how many emails the agent filed — measures activity, not correctness. It would have looked excellent during the week the two questions disappeared.

The metric that catches this failure: sample archived threads and count how many were awaiting a reply from the user. A dozen threads a week, checked by hand, takes a few minutes and directly measures the harm you care about.

⚠️ For anything that files, closes, hides, or deletes, the useful measurement is always about what was wrongly removed — never about volume processed. Volume goes up when the agent is being too aggressive, which is exactly the failure.

💡 The general lesson: pick the right unit

The specific bug was about email. The shape of it turns up everywhere:

• Classifying a file when the meaningful unit is the change across several files.
• Judging a row when the record spans several rows.
• Reviewing a message in a support conversation rather than the case.
• Assessing a log line instead of the incident it belongs to.

The rule of thumb: the unit you classify should be the unit you act on. The agent was acting on threads — archiving them — while judging messages. Any time those two differ, the decision is being made on partial information, and it will look correct most of the time, which is what makes it hard to spot.

The takeaway

Before improving a prompt, check what the agent is actually looking at when it decides. If it acts on a whole thing but only sees a piece of it, no amount of prompt tuning will fix the gap. Assemble the full unit, ask a question that forces the model to examine all of it, keep the action reversible while you build confidence, and measure what was wrongly removed rather than how much got processed.