Your Agent Retried the Tool Call. The Customer Got Charged Twice.
A timed-out tool call is not a failed one — it is an unknown one. Here is why "just retry" quietly doubles real-world side effects, and where to put the idempotency check so it cannot.
A refund tool call timed out after thirty seconds. The agent did exactly what it was built to do: it read the error, decided the operation had not happened, and called the tool again. The second call succeeded. So had the first one — the response just never made it back.
This is the most boring catastrophic bug in agentic systems, and it has almost nothing to do with the model. The guardrail that prevents it is not a better prompt. It is a decision about what counts as the unit of work.
🔍 A timeout is not a failure
Every call that leaves your process has three possible outcomes, not two:
- It succeeded and told you.
- It succeeded and the answer was lost — timeout, dropped connection, load balancer reset, your process died mid-flight.
- It never happened.
The tool result your agent sees collapses states 2 and 3 into one string: Error: request timed out. The model has no way to tell them apart, because the information genuinely is not there. Asking it to be more careful is asking it to guess.
That matters because a well-built agent loop is designed to recover from errors. Recovery and duplication are the same behavior viewed from two sides.
The four places a duplicate is born
Only one of them involves the model at all.
- Transport retry. Your HTTP client or vendor SDK retries on 5xx and connection errors by default. The agent never sees the first attempt.
- Agent-loop retry. The model reads the error in the tool result and calls the tool again. This is the visible one.
- Process restart. The run crashes, a supervisor resumes it from the last checkpoint, and the step that was in flight at crash time is replayed.
- Duplicate dispatch. A queue redelivers an unacknowledged message, or a user clicks the button again, and two workers act on the same task.
Because three of the four happen below or outside the reasoning loop, "instruct the agent not to retry" is not a guardrail. It closes one of four doors.
Key the effect, not the attempt
The fix is an idempotency key: a value that identifies the thing the agent is trying to accomplish, so that every attempt at it carries the same label and the second one can be recognized and dropped.
The whole trick is in how the key is derived.
# ❌ New key on every attempt. Deduplicates nothing.
key = str(uuid.uuid4())
# ✅ Same key for every attempt at the same intended effect.
key = sha256(f"{run_id}:{step_id}:{canonical_json(args)}").hexdigest()
Derive it from intent — the run, the step, and the normalized arguments — and it survives a transport retry, a model retry, and a resume-from-checkpoint alike. Generate it per attempt and you have written a very expensive no-op.
One subtlety worth getting right: canonicalize the arguments before hashing. Sorted keys, fixed number formatting, no timestamps, no request ids. If the model regenerates the arguments on its retry and re-serializes them in a different order, an intent-derived key still changes and the dedupe silently stops working.
Three places to put the check
In the provider. Many payment, messaging and provisioning APIs accept an idempotency key on the request and will return the original response for a repeat. When the target supports it, use it — the check lives on the same side as the effect, which is the only place it can be perfectly correct.
In your tool wrapper. When the target has no such support, keep an effect ledger of your own and write to it before the call, not after:
insert (key, state=pending) -- fails if the key already exists
call the tool
update (key, state=done, result) -- store the response
A second attempt hits a unique-constraint violation on the insert and returns the recorded result instead of calling anything.
In the target's data model. Sometimes the cleanest key already exists — one refund per charge, one welcome email per account. A unique constraint on the real-world invariant needs no ledger at all.
⚠️ The pending row is the hard part
Write-ahead ledgers have one nasty state: a row stuck at pending because the process died between the insert and the update. The next attempt now knows an attempt was made, and still does not know whether it landed.
Do not treat pending as "safe to retry." Treat it as "go find out." Every keyed tool needs a reconciliation path — query the provider for an operation carrying this key, or look for the effect itself — plus a timeout after which an unreconciled key escalates to a human rather than resolving itself in whichever direction is convenient.
Classify your tools before you write any of this
Most tools do not need a key. Sort them once, and the work shrinks:
| Tool shape | Repeat costs | What it needs |
|---|---|---|
Read-only (get_order, search_docs) |
Nothing | Retry freely |
Overwriting write (set_status("closed")) |
Nothing | Retry freely |
Accumulating write (append_note, increment_credits) |
Silent data corruption | Idempotency key |
Irreversible external (charge, send_email, deploy) |
Money, trust, an outage | Key and reconciliation |
The bottom row is the only one that should ever block a release. An agent that can retry an unkeyed call in that row is one dropped connection away from doing the thing twice — and no amount of eval coverage on the happy path will catch it, because the happy path is where the response comes back.
The takeaway
Retry policy looks like error handling, so it gets designed as error handling: catch, back off, try again. In an agent with real-world tools it is something else. Every retry rule you write is a claim about which side effects are safe to duplicate, and that claim belongs in the tool definition — next to the schema, decided once by the person who knows what the tool actually does — not in the loop that discovers the error at three in the morning.