Cached or Re-Called? Giving Every Agent Tool a Staleness Budget

An on-call agent picks up an incident, calls get_service_health("checkout"), and gets back degraded. It then spends four minutes pulling logs, correlating a recent deploy, and drafting a remediation. Finally it calls rollback_deployment.

The health reading it acted on was four minutes old. The service had recovered on its own three minutes earlier.

The agent did nothing wrong by its own logic. Every fact it used was true at the moment it was read. The bug is that nobody ever decided how long a tool result stays usable.

Two failure modes get collapsed into one question

"Should we cache tool calls?" hides two opposite problems, and answering it globally guarantees you make one of them worse.

• Redundant re-calls. The agent asks for the service inventory three times in one run because the earlier answer scrolled out of its working attention. You pay latency, tokens, and rate-limit budget for information you already had.
• Stale reads. The agent asks once, holds the answer for the rest of the run, and commits an action against a world that has moved.

A single global TTL trades one for the other. Set it to zero and every run re-fetches immutable data. Set it to five minutes and your incident agent rolls back a healthy service. The unit of the decision is not the cache — it's the tool.

Give every tool a staleness budget

A staleness budget is the maximum age at which a result is still trustworthy at the moment of use, not at the moment of fetch. That distinction is the whole point. A value fetched two seconds into a run and consumed at minute four is a four-minute-old value, no matter how fresh the fetch felt.

Three questions set the budget.

1. How fast does the underlying value change?

Sort every tool into three volatility classes:

• Immutable — a merged commit's diff, a closed ticket's creation timestamp, a released version's changelog. These can be cached for the run and well beyond it.
• Slow — service ownership, on-call rotation, a customer's plan tier, region topology. Minutes to hours.
• Fast — queue depth, pod health, current price, seat availability. Seconds, or don't cache at all.

Most teams have more immutable tools than they realize. Those are free wins with no correctness cost.

2. What is the agent going to do with the value?

There are two grades of consumption, and they deserve different budgets from the same tool.

Orientation — the value steers the next investigative step. A stale reading here costs you one wasted tool call and the model usually self-corrects.

Commitment — the value is a precondition for a write, a spend, a notification, or a rollback. A stale reading here is the incident above.

This gives the single highest-value rule in the whole design:

Any value that gates a write is re-read immediately before the write, regardless of what its cache budget says.

That re-read is cheap relative to what it protects. It also naturally produces a check the model can act on: if the fresh reading contradicts the plan, the plan is wrong, and the agent should say so rather than proceed.

3. What does a re-call actually cost?

A cheap idempotent GET against your own service? Just call it again — the caching complexity isn't worth it. A third-party API with a hard per-minute quota, or a warehouse query that takes six seconds? Caching earns its keep, and the budget should be a deliberate number rather than an accident of how the loop happens to be written.

Writing it down

The artifact is small. One entry per tool, next to the tool definitions:

TOOL_CACHE = {
    "get_commit_diff":     {"ttl": None},                              # immutable
    "get_service_owner":   {"ttl": 3600},
    "list_open_incidents": {"ttl": 60},
    "get_service_health":  {"ttl": 15, "revalidate_before_write": True},
    "get_queue_depth":     {"ttl": 0},                                 # never cache
}

And the wrapper honours the purpose the caller declares:

def call_tool(name, args, purpose="orient"):
    spec = TOOL_CACHE[name]
    hit  = cache.get((name, args))

    fresh_required = (
        purpose == "commit" and spec.get("revalidate_before_write")
    )
    if hit and not fresh_required and not expired(hit, spec["ttl"]):
        return annotate(hit.value, age=now() - hit.fetched_at)

    value = invoke(name, args)
    cache.set((name, args), value)
    return annotate(value, age=0)

Hand the model the age, not just the value

The quiet mistake in most agent caches is returning a cached result formatted identically to a fresh one. That strips the model of any chance to reason about it.

Return the age in the tool result instead:

{ "value": "degraded", "observed_seconds_ago": 214 }

Models handle this well. Given a number that looks large relative to the action they are about to take, they will re-check — and when they don't, you now have an explicit trace of a decision made on a 214-second-old reading, which is a thing you can review.

🔍 What to watch in traces

Two metrics tell you whether the budgets are real:

• Age-at-use. For every tool result the model actually cited in a decision, log how old it was. Look at the p95 for tools that gate writes. If the intended TTL is 15 seconds and p95 age-at-use is three minutes, your cache key or your revalidation path is broken.
• Duplicate (tool, args) pairs per run. These are candidates for caching — but read a few before assuming waste. Sometimes the repetition is deliberate polling, and caching it would break the loop.

The cheap default

If a per-tool table feels like too much ceremony for where you are, three rules capture most of the value:

✅ Cache immutable and slow-moving lookups for the length of the run. ✅ Re-read anything that gates a write, right before the write. ❌ Don't apply one global TTL across every tool.

The framing that helps most: an agent's tool results are not facts, they are observations with timestamps. The moment you treat them as timeless, the agent's reasoning quality stops being the thing that determines whether its actions are correct.