Your Agent Works in the Demo Because You're Watching: Six Failures That Only Appear at 3 A.M.
A live demo isn't a smaller version of production — it's a different environment with a human silently patching over every gap. Here are the six failure classes that only wake up once that human goes to bed.
The demo ran clean. Fourteen steps, a tidy summary, applause. Two weeks later the same agent runs on a cron trigger at 03:00 and produces a report claiming the pipeline is healthy, for a pipeline that has been down since Tuesday.
Nothing regressed. The agent is doing exactly what it did in the demo. What changed is that nobody is in the room, and the person in the room was load-bearing.
A watched run and an unwatched run are not the same environment. The watched one has a human silently supplying fresh credentials, sane inputs, a working terminal, patience with a slow step, and a finger on the kill switch. Strip those out and six distinct failure classes surface — none of which a demo can reveal, because the demo's whole shape is the thing hiding them.
⚙️ 1. The question asked into an empty room
The most common one, and the least dramatic. The agent hits ambiguity and does what it was trained to do: asks. Which of the three staging environments did you mean?
In the demo you answered in four seconds and forgot it happened. Unattended, that question goes to stdout, nobody reads it, and the run either hangs until a timeout kills it or — worse — the harness treats the unanswered turn as an empty input and the agent invents an answer to its own question.
Test for it directly: run the agent with stdin closed and any human-input tool stubbed to return NO_HUMAN_AVAILABLE. Every path that reaches that stub is a path that will hang or hallucinate at 3 a.m. Decide for each one whether it should default, defer, or abort — but decide.
2. Approval gates that quietly degrade into approvals
Related, and nastier, because it fails in the unsafe direction. A confirm-before-write gate is a real control when a person is there to say no. Unattended, teams reach for a --yes flag, or an auto-approve policy "just for the scheduled run," and the gate becomes a log line that says a gate happened.
The gate was never the prompt. The gate was the human. If you remove the human you have to replace the check, not the ceremony — an allowlist of writable targets, a row-count ceiling, a dry-run diff compared against expected shape. Something that can return false with nobody present.
❌ if (interactive) confirm() else proceed()
✅ if (interactive) confirm() else assertWithinPolicy()
3. State that was fresh only because you just made it fresh
Before every demo, somebody logs in. That login seeds an OAuth token, a browser session, a warm cache, a kubectl context. The agent never exercises the expiry path because in a watched run nothing has had time to expire.
Unattended runs live on the other side of that boundary permanently. The token is nine hours old. The session cookie died at midnight. The cached schema is from last week's deploy and the column got renamed.
The signature failure is not a clean auth error — it's a 401 body rendered as HTML, handed to the agent as a tool result, and reasoned over as if it were data. The agent reports what the login page said.
🔍 Cheap diagnostic: run the agent once with every credential deliberately expired. You are not testing whether it succeeds. You are testing whether it notices. An agent that fails loudly on expiry is fine; an agent that summarises the login page is a liability.
4. Two runs in the same room
Your demo ran once. The scheduler doesn't know that. If run N takes forty minutes and the cron fires every thirty, run N+1 starts on top of it, and now two agents are editing the same ticket, appending to the same file, or both claiming the same work item.
This never appears in a demo because a demo is a single invocation with a human waiting for it to end. Unattended, the overlap is the default state the moment latency drifts — and agent latency drifts constantly, because it's a function of how many tool calls the model decides to make.
Pick one: a lock with an owner and a lease, or an idempotency key on every side effect. Lock alone deadlocks when a run dies mid-flight; idempotency alone still burns double the tokens. Most schedules want both.
5. The input distribution you unconsciously curated
You picked the demo ticket. You picked it because it was representative — meaning it was legible. It had a clear description, one attachment, no reply chain, English, under a page.
Production hands the agent the empty ticket, the 400-message thread, the one that's just a screenshot, the duplicate, and the one where the customer pasted a stack trace into the subject line. None of these is exotic. They were filtered out by your judgement, which was doing feature selection for free.
The fix isn't more test cases picked the same way. It's to stop picking: sample 50 real inputs at random from the last month, run them unattended, and read every trajectory. The ones you'd never have chosen are the ones worth reading.
6. Failure that has nowhere to go
In a demo, failure output is observed by definition — you're looking at it. Unattended, an agent that fails has to actively report failure, and most agents are better at summarising than at escalating.
The specific shape to watch for: the agent encounters a broken step, reasons around it, and produces a confident, complete-looking output that silently omits the broken part. No exception is raised. The exit code is 0. The report says "3 of 3 checks passed" because the fourth check never ran.
This is why unattended runs need an assertion the agent does not control — a post-run check outside the loop that verifies the work actually landed. Row count changed. File written. Ticket status moved. If the only evidence a run succeeded is the agent's own summary of it, you have a narrator, not a monitor.
💡 The unwatched-run checklist
Before any agent goes on a schedule, run it once with all of this true at the same time:
- stdin closed, human-input tools stubbed to refuse
- every credential expired or near-expiry
- a second instance started 60 seconds behind it
- inputs sampled at random, not chosen
- one dependency deliberately broken
- success verified by something outside the agent
If it survives that, you've tested the thing you're actually deploying. If you've only ever watched it work, what you've validated is the demo — and the demo has a human in it.