How to Implement Zero-Downtime LLM Failover for OpenAI and Anthropic

The Cost of an OpenAI Outage

When an upstream LLM provider like OpenAI or Anthropic experiences an outage, a 503 error or a 30-second timeout can cripple your AI application. For production systems, standard exponential backoff retries are insufficient—they only delay the inevitable crash.

To achieve high availability, you must implement an LLM circuit breaker with automatic provider failover.

The Circuit-Breaker Pattern for AI Gateways

A circuit breaker monitors the health of your primary provider (e.g., OpenAI). If the error rate exceeds a specific threshold (e.g., 30% failure over 10 seconds), the breaker "trips" into an Open state.

Once tripped, the AI Gateway instantly routes all incoming traffic to a standby provider (e.g., Anthropic Claude or Google Gemini) seamlessly. The client application never knows the primary provider went down.

Building Intelligent Fallback Routing

Intelligent failover requires semantic mapping between providers. An AI gateway must automatically translate OpenAI-formatted messages into Anthropic's format on the fly.

// Pseudocode for Gateway Routing
async function routeLLMRequest(prompt) {
  if (circuitBreaker.isOpen('openai')) {
    // OpenAI is down, fallback to Anthropic
    const anthropicPayload = translateToAnthropic(prompt);
    return await fetchAnthropic(anthropicPayload);
  }
  
  try {
    return await fetchOpenAI(prompt);
  } catch (error) {
    if (is503(error)) circuitBreaker.recordFailure('openai');
    throw error;
  }
}

Streaming Failover Complexity

Handling failovers during Server-Sent Events (SSE) streaming is notoriously difficult. If the connection drops mid-stream, the proxy must catch the error, open a connection to the fallback provider, and append the remaining stream chunks without breaking the client's parser. This is a core feature of the Selixes Gateway.

Conclusion

Do not wait for the next global LLM outage to realize your app needs high availability. Implement an AI gateway with native, cross-provider circuit breaking to guarantee zero-downtime AI deployments.

See It in Action

Selixes implements everything described in this article — circuit breaking, session budgets, local edge fallback, and private VPC deployment.

More Articles