The Invoice Tool Timed Out, So the Billing Agent Called It Again. The Customer Was Charged Twice. A Timeout Is Not a Failure.

A billing agent calls charge_customer and gets back a timeout. It does what any helpful agent would do: it tries again. The second call succeeds, the agent reports "payment collected", and two days later the customer emails a screenshot showing two identical charges.

Nothing in that run looks like a bug. The model followed its instructions, the tool returned honest errors, and the retry was the right move by every rule the agent had been given. The flaw sits in one assumption nobody wrote down: that a timeout means the action did not happen.

🔍 Walking through the run

Here is the trace, simplified:

step 4  call charge_customer(customer_id="c_812", amount=149.00)
        -> error: "upstream timeout after 30s"
step 5  thought: "The charge failed. I'll retry once."
step 6  call charge_customer(customer_id="c_812", amount=149.00)
        -> ok: {"charge_id": "ch_9f2", "status": "succeeded"}
step 7  reply: "Payment of $149.00 collected."

What actually happened at step 4: the payment provider received the request, processed the charge, and was slow to respond. The tool wrapper gave up waiting after 30 seconds and returned an error. The charge from step 4 was real. It just never reported back.

So the agent's world model after step 4 ("no charge exists") was wrong, and step 6 acted on it.

Three outcomes, not two

Most tool wrappers present a call as binary: it worked, or it failed. Any call that crosses a network actually has three outcomes:

1. Succeeded, and you know it.
2. Failed before doing anything, and you know it (bad input, auth rejected, connection refused).
3. Unknown. The request may or may not have taken effect. Timeouts, dropped connections, and 502s from a proxy all land here.

The third bucket is the dangerous one, because it is usually reported with the same shape as the second. An agent reading error: upstream timeout has no way to tell "nothing happened" from "something happened and I didn't hear about it." Language models also lean towards the optimistic reading that lets them make progress, and "it failed, so try again" is exactly that reading.

For read-only tools this doesn't matter. Searching twice costs a few tokens. For tools that move money, send messages, create tickets, or provision infrastructure, retrying from the unknown bucket duplicates the action.

❌ The fixes that don't work

"Tell the agent not to retry." Now a genuine connection error leaves the customer uncharged and the agent stuck. You traded a duplicate for a missed action, and the prompt rule will be forgotten the first time a user says "please try again."

"Have the agent check before retrying." Better, but it relies on the model choosing to call list_charges first, matching the right record, and handling the case where the first charge shows up a few seconds late. That's a lot of judgment to put on the path where money moves.

"Raise the timeout." Fewer timeouts, same bug. The unknown bucket gets smaller but never empty.

✅ The fix: make the retry safe, not rare

The robust answer lives in the tool, not the prompt. Give every side-effecting call an idempotency key: a stable identifier for this intended action, so that sending it twice has the same effect as sending it once.

Many payment and messaging APIs accept a key like this natively. If yours doesn't, your tool wrapper can keep its own record of keys it has already executed.

def charge_customer(customer_id: str, amount: float, intent_id: str):
    # intent_id identifies the action, not the attempt.
    key = f"charge:{customer_id}:{intent_id}"

    prior = ledger.get(key)
    if prior and prior.status == "succeeded":
        return {"status": "already_done", "charge_id": prior.charge_id}

    try:
        result = payments.charge(customer_id, amount, idempotency_key=key)
        ledger.put(key, result)
        return {"status": "succeeded", "charge_id": result.id}
    except Timeout:
        return {
            "status": "unknown",
            "message": "The charge may have gone through. "
                       "Retrying with the same intent_id is safe; "
                       "do not use a new intent_id.",
        }

Three details carry the weight here.

The key comes from the intent, not the attempt. If the agent invents a fresh random key on each call, you've built nothing. The key should be derived from something stable: the invoice id, the order id, the ticket the agent is working on. A good pattern is to have an earlier step (say, create_invoice) return the id, and require the charge tool to take it.

Unknown gets its own status. The tool result no longer says "error". It says unknown, and it tells the agent exactly what is safe to do next. You're spending a few tokens in the response to steer the model's next move instead of hoping it guesses.

Duplicates return success, not an error. When the retry hits a key that already succeeded, the tool answers already_done with the original charge id. If it returned an error, the agent would try to work around it, and that is how you get a third charge with a "fixed" key.

A quick audit for your own toolset

Go through every tool your agent can call and sort them:

• Read-only (search, get, list): retry freely.
• Naturally idempotent (set status to "closed", upsert by id): retry freely, but check the tool really is idempotent and not "append a status change event."
• Side-effecting and not idempotent (charge, send, create, post, deploy): needs a key, an unknown status, and an already_done response before an agent is allowed to call it.

That last group is usually small. In a typical billing or support agent it might be a handful of tools. Those are the ones that need the extra work, and they are also the ones where a duplicate turns into a refund request or an apology email.

What to watch in traces

Once the fix is in, two signals are worth alerting on. Calls that return unknown tell you how often the agent is operating blind, and a spike usually points at a slow upstream. Calls that return already_done tell you the safety net actually caught something. A steady trickle is fine and expected. Zero, forever, may mean the agent is generating a new key per attempt and the net has a hole in it.

A timeout only tells the agent that it stopped waiting. It says nothing about whether the charge went through. Build the tool so the agent never has to guess which it was, and retrying is safe whichever one it turns out to be.