The Refund Tool Timed Out, the Agent Retried, and the Customer Got Paid Twice. Make Every Write Tool Idempotent.

A support agent calls issue_refund(order_id="A-1042", amount=49.00). The payment provider takes eleven seconds to answer, the tool wrapper gives up at ten, and the agent sees Error: request timed out. It does what any sensible operator would do with that information: it tries again. The second call succeeds. So did the first one. The customer is refunded twice, and nothing in the transcript looks wrong.

This post takes that incident apart, because no single component in it is broken. The agent behaved reasonably, the provider behaved correctly, the timeout was a sensible default. The bug lives in the tool's contract, and that is where it has to be fixed.

🔍 The autopsy: three reasonable decisions that add up to a double charge

Walk the incident step by step and each decision holds up on its own.

1. The tool wrapper set a timeout. Correct. An agent loop that can hang forever on one call is worse.
2. The timeout was reported as an error. Also correct, from the wrapper's point of view. It did not get a response.
3. The agent retried on error. Exactly what the system prompt asked for: "If a tool call fails, try again once before escalating."

The flaw is in step 2's wording. A timeout is not a failure. It is an unknown outcome. The request may never have arrived, may have arrived and failed, or may have arrived and succeeded with the response lost on the way back. The tool collapsed three different states into one word, "error", and the agent acted on the word.

A human support rep in the same seat would probably check the order history before clicking refund again. An agent will only do that if the tool tells it the state is uncertain, or if the tool makes the retry harmless.

❌ The fixes that don't hold

"Tell the agent not to retry refunds." This moves the problem instead of solving it. Now a genuine transient failure (the request never left your network) leaves the customer with no refund and a confused escalation. It also has to be remembered for every write tool anyone adds later.

"Tell the agent to check before retrying." Better, but it relies on the model choosing to call a second tool at the right moment, every time, under a prompt that may be long and busy. Instructions like this decay quietly as the prompt grows.

"Raise the timeout." This narrows the window without closing it. Networks drop responses at any duration.

All three put the safety of a money-moving operation in the model's judgment. The model's judgment is the part of the system you can least audit.

✅ The fix: an idempotency key the agent cannot get wrong

Make the write safe to repeat. The standard mechanism is an idempotency key: a value sent with the request so the receiving side can recognise a repeat and return the original result instead of acting twice. Many payment APIs support this natively; if yours does not, you can enforce it in your own tool layer with a small table.

The detail that matters for agents is where the key comes from. If the model generates it, it can generate a fresh one on retry and you are back where you started. Derive it from the intent, inside the tool, from arguments that identify the business action:

def issue_refund(order_id: str, amount: float, reason_code: str) -> dict:
    key = sha256(f"refund:{order_id}:{amount:.2f}:{reason_code}").hexdigest()

    existing = refunds_log.get(key)
    if existing:
        return {"status": "already_done", "refund_id": existing.refund_id,
                "note": "This refund was issued earlier. Do not retry."}

    try:
        result = payments.refund(order_id, amount, idempotency_key=key)
    except Timeout:
        return {"status": "unknown",
                "note": "Outcome unknown. Calling again with the same arguments is safe."}

    refunds_log.put(key, result.refund_id)
    return {"status": "done", "refund_id": result.refund_id}

Now the retry the agent was always going to make is harmless. The same arguments produce the same key, the provider (or your log) recognises it, and the second call returns the first call's result.

Two things changed besides the key:

• Timeouts return unknown, not error. The agent gets an accurate description of the state, plus an explicit statement that repeating is safe.
• Replays return already_done with the original id. The agent can report the real refund id to the customer instead of inventing one or apologising for a failure that did not happen.

⚠️ Where intent-derived keys need care

A key built from arguments has one real weakness: two legitimate identical actions look like one. If a customer is genuinely owed two separate 49.00 refunds on the same order, the second will be swallowed.

Handle it by putting the distinguishing fact into the arguments, not by loosening the key. A line_item_id or a support ticket_id usually does the job. Ask this question of every write tool: what makes two calls the same business action? Whatever answers it goes into the key.

Also decide how long a key lives. A refund key should outlive any plausible retry window, which for an agent that can be paused for human review might be days, not seconds.

💡 A checklist for every tool that writes

Run each write tool through these before an agent can call it:

• Can a response be lost after the action happened? If the tool talks over a network, yes.
• What identifies the business action? Those fields form the idempotency key, computed inside the tool.
• Does the model ever supply the key? It should not.
• What does a timeout return? It should say unknown and state whether a retry is safe.
• What does a replay return? The original result, marked as a replay.
• Does the tool description say it is safe to retry? One sentence here saves a paragraph of prompt instructions.
• For tools you cannot change (a third-party API with no key support): add a lookup before the write inside the wrapper, so the check happens in code rather than in the model's plan.

Sending email, creating tickets, posting to a channel and booking a meeting all have the same shape as the refund. They are less costly to duplicate, so the bug survives longer before anyone notices it.

The point

Agents retry. It is the correct reflex for a system that cannot see across the network, and prompting it away trades duplicate actions for dropped ones. The durable fix is a tool that describes uncertainty honestly and makes repetition safe by construction. Once the refund tool is idempotent, "try again once" stops being a risky instruction and becomes a harmless one.