Retry, Reroute, or Hand Back: Budgeting an Agent's Second Chances

Your scheduling agent tries to book a meeting. The calendar API returns a 429. The agent retries. Retries again. Backs off, retries a third time, a fourth, a fifth — and forty seconds later tells the user "I wasn't able to schedule that." The user is now worse off than if the agent had failed instantly: they waited, they paid for the tokens, and they still have to open the calendar themselves.

That is the failure mode worth designing against. Not "the tool call failed" — tool calls fail constantly and that is fine — but an agent that spends its entire budget proving it cannot do something, and then hands back nothing usable.

⚙️ A failed tool call has three legal responses, not one

Most agent loops implement exactly one: try again. The other two matter more.

• Retry — same tool, same arguments, later. Correct only when the failure was about timing.
• Reroute — different tool, different arguments, or a narrower version of the goal. Correct when the failure was about this particular path.
• Hand back — stop, and return control to the human with everything learned so far. Correct when the failure was about information the agent does not have.

A retry loop that never considers the other two branches will hammer a permanently broken door. Ask a scheduling agent to book a room it lacks permission to book, and no number of attempts changes the answer — but the loop does not know that unless someone taught it to look at why it failed.

🔍 Classify the error before you count attempts

The attempt counter is the second decision, not the first. The first is: what kind of failure is this?

Transient. Rate limits, timeouts, 5xx, connection resets. The world will probably be different in two seconds. → Retry, with backoff, small cap.

Deterministic. 401, 403, 404, schema validation errors, "calendar not found." Nothing about waiting changes the outcome. → Never retry. Reroute if there is a plausible alternative path, otherwise hand back immediately.

Ambiguous. The call succeeded technically but the result is unusable: an empty availability list, a conflict the agent cannot resolve, three people named "Sam" in the directory. → Reroute once with a narrower query, then hand back. This is the category that eats budgets, because it never looks like an error.

A useful rule of thumb: ❌ if the same call with the same arguments would produce the same result, retrying is just a slow way of failing. ✅ retry only when time itself is the variable you are changing.

💡 Give the agent a budget it can see

Attempt counts alone are a weak budget, because three retries against a slow API can cost more wall-clock time than fifteen against a fast one. Track three ceilings at once, and expose the remaining headroom to the model:

budget = {
    "attempts_left": 3,        # per tool call
    "seconds_left": 20,        # for the whole user-facing task
    "tokens_left": 8_000,      # what's left of the turn
}

def on_tool_failure(err, budget):
    kind = classify(err)                     # transient | deterministic | ambiguous
    if kind == "deterministic":
        return "hand_back"
    if budget["attempts_left"] <= 0 or budget["seconds_left"] < 5:
        return "hand_back"
    if kind == "ambiguous":
        return "reroute"                     # one narrower attempt, then out
    return "retry"

The seconds_left < 5 check is the part teams skip and then miss. An agent that starts a retry it cannot finish inside the user's patience window has converted a fast failure into a slow one for nothing.

Put the same numbers in the model's context, not just in the harness. An agent told "you have 1 attempt and roughly 8 seconds left" makes visibly different choices from one told nothing — it stops exploring and starts summarizing. Budget awareness is a prompt feature, not only a control-flow feature.

The handback is a deliverable, not an apology

If the agent is going to stop, the stop has to be worth something. "I couldn't schedule that meeting" is a wasted turn. Compare:

I couldn't book the room. The calendar API accepted the 2–3pm slot but rejected the room booking with a permissions error on Conference Room B — that room appears to need approval from facilities. Two options that don't need approval: Room D is free 2–3pm, or I can book the same slot with no room and add a video link. Which do you want?

Everything expensive the agent learned is preserved: the slot is confirmed available, the blocker is named, the failure is attributed to a specific cause, and the user's next action is one word long. That handback cost the same tokens as the useless one. The difference is entirely in what the loop was instructed to carry out with it.

A good handback contains four things:

1. What succeeded — so the work is not repeated.
2. What failed, specifically — the tool and the reason, not "an error occurred."
3. Why the agent stopped — budget exhausted, or a wall it cannot pass.
4. The smallest decision the human can make to unblock it — ideally a choice between two concrete options.

Where to set the dials

There is no universal number, but there is a defensible starting point: one retry for transient errors on an interactive task, three for a background job, zero for anything deterministic. Interactive agents should spend their budget on getting a good handback out fast; background agents can afford patience because nobody is watching a spinner.

Then instrument it. The signal to watch is not the failure rate — it is the ratio of retried attempts to successful recoveries. If your agent retried two hundred calls last week and forty of them eventually succeeded, the budget is roughly earning its keep. If four succeeded, you are paying for a hundred and ninety-six slow failures, and the fix is not a better backoff curve. It is a lower ceiling and a better handback.

An agent that fails fast and explains itself precisely is more useful than one that fails slowly after trying everything. Persistence is only a virtue when the door can actually open.