Designing fallback that actually survives an outage

Every serious AI application has a fallback story. Most of those stories are fiction. The chain is written the week the provider has its first outage, committed with a note that says "TODO: test this", and never touched again until the next incident proves it doesn't work.

The instinct is to treat fallback as plumbing: primary fails, try secondary, done. But the interesting questions are all product questions.

• Is a slower answer better than no answer, or is your use case latency-critical enough that a slow fallback is itself a failure?
• Is a cheaper, weaker model an acceptable degradation, or does dropping quality silently break downstream code that expects structured output?
• Does a mid-stream failure (tokens already delivered to the user) get retried, or continued, or abandoned?

None of these have a universal right answer, which is exactly why they can't be left to a default.

In practice, one fallback chain is rarely enough, because different paths have different tolerances.

Interactive chat   → fast primary → fast alternative → cached/canned response
Batch extraction   → cheap primary → cheap alternative → retry with backoff
Critical pipeline  → pinned model → pinned alternative → hard fail + alert

The critical pipeline is the counterintuitive one: sometimes the right fallback is to fail loudly rather than to silently substitute a model whose output your downstream code wasn't validated against.

A fallback that changes the shape of your output without telling anyone is not a safety net. It is a second, quieter bug.

You cannot verify a fallback chain by reading it. You verify it by breaking the primary on purpose and watching what happens, in staging, on a schedule, as a standard part of your deploy checks. If your gateway supports it, force a provider into a failed state and confirm the chain does what the config claims.

The teams whose fallbacks work are not smarter. They are the ones who treat "the primary is down" as a routine, tested state rather than an emergency they hope to handle well when it arrives.