Graceful Degradation When the Model Provider Is Down

Model providers have incidents. Rate limits tighten without warning, latency spikes, a region has trouble. If your agent is a feature inside a larger product, the question is what the rest of the product does during those minutes — and the default answer, an error where a useful feature used to be, is rarely the best available.

Degradation levels, not on/off

Design a ladder rather than a binary. Each rung is worse than the one above and better than failure:

1. Normal. Full agent, preferred model.

2. Fallback model. Same agent, different provider or a smaller model of the same family. Requires your prompts to work on both, which means testing them there — a fallback that has never been exercised is a hypothesis.

3. Reduced scope. Fewer turns, no optional verification steps, smaller retrieval. Runs complete with less thoroughness, and the output says so.

4. Cached and deterministic paths only. Serve previously computed answers where they apply, and run the rule-based path for requests that don't need judgment. If a deterministic filter handles most of your volume, most of your volume survives the incident untouched.

5. Queue for later. Accept the request, tell the user it'll be processed shortly, run it when capacity returns. Correct for anything not interactive, and much better than a failure the user has to remember to retry.

6. Honest failure. Clear message, clear expectation, no silent partial results.

Most products can implement 4, 5, and 6 cheaply, and those cover the bulk of the value.

The fallback that actually works

A second provider isn't a drop-in. Behavior differs enough that a fallback path nobody tests will fail in its own way.

• Test the fallback continuously. Route a small percentage of production traffic to it always, so you know it works and roughly how well. A fallback exercised only during incidents is exercised for the first time during an incident.
• Keep prompts per provider, not shared. Shared prompts underperform on at least one of them.
• Assert your safety invariants on both. A fallback that's weaker against injection is a security hole that opens exactly when you're distracted.
• Know the quality delta, and decide in advance whether it's acceptable for which features.

⚠️ For agents holding destructive tools, consider whether the fallback should be read-only. Running your most consequential actions on a less-tested configuration during an incident is a way to turn one problem into two.

Detect fast, recover carefully

Circuit-break on the provider. After a threshold of failures or timeouts, stop attempting and degrade immediately. Otherwise every request waits for a timeout, and your own system's queues back up behind an upstream problem — the failure amplifying itself.

Distinguish the failure types. Rate limiting, timeouts, and errors want different responses: rate limits want backoff and shedding, timeouts want the circuit breaker, errors want the fallback.

Recover gradually. When the provider returns, ramping all traffic back at once against a recovering service can knock it over again. Ramp over minutes.

Tell the user something true

The degraded experience should be legible:

• ✅ "Using a faster model right now — answers may be less detailed."
• ✅ "Queued; you'll get this within the hour."
• ❌ Silently returning a worse answer as though it were normal.
• ❌ A generic error with no expectation set.

The silent-degradation option is tempting and corrosive. Users notice quality changes and, absent an explanation, attribute them to the product being unreliable rather than to a bad afternoon.

🔍 Rehearse it

Once, deliberately, in a non-production environment: block the provider and watch.

• Does the circuit breaker trip within seconds?
• Does the fallback produce acceptable output?
• Do queued requests actually run when service returns?
• Do in-flight runs fail cleanly, with recorded state, or leave locks held?
• Does the user-facing message make sense?

Every gap this finds is one you'd otherwise find with an audience.

The takeaway

Build a ladder — fallback model, reduced scope, cached and deterministic paths, queue, honest failure — rather than a binary. Exercise the fallback continuously in production so it's known to work, circuit-break quickly so upstream trouble doesn't back up into your own queues, ramp recovery gradually, and tell users plainly when they're getting the degraded version. Then rehearse it once, because that's the difference between a plan and a document.