Dynamic Latency-Cost Routing

Learn how the gateway analyzes token metrics and provider response times in real-time to lower your bill without sacrificing quality.

What is Dynamic Routing?

Normally, choosing which LLM model to call (e.g., GPT-4o, Claude 3.5 Sonnet, or Llama 3) is decided at development time in your application code. This static routing introduces two critical inefficiencies:

Latency Spikes: If your chosen provider suffers a sudden load spike, your application becomes sluggish for all users.
Overpaying for Simple Queries: Standard queries (like formatting, classification, or validation) are routed to expensive models when cheaper models could achieve the same outcome.

Selixes resolves this by moving the routing decision to the **request-transit layer**. The gateway acts as a smart broker, assessing incoming prompts and routing them dynamically.

How Latency Metrics Are Tracked

The gateway maintains a rolling window of latencies for all registered upstream models. This data is collected passively from production transits and aggregated in a local Redis cache:

Time-to-First-Token (TTFT): Critical for streaming endpoints; measures the delay before response text starts generating.
Total Duration: Measures the entire latency of the HTTP request from start to completion.
EMA Algorithm: The gateway uses an Exponential Moving Average (EMA) to give more weight to recent requests, allowing it to adapt to provider outages or recovery within seconds.

Configuring Routing Policies

You can declare your routing preferences at request-time using gateway headers. Selixes supports three primary strategies:

Policy Mode	Header Value	Description
Lowest Cost (Arbitrage)	cost-arbitrage	Routes requests to the cheapest available model capable of handling the task complexity (checks context size and token costs).
Lowest Latency	lowest-latency	Routes requests to the provider currently reporting the lowest Time-to-First-Token (TTFT) in the moving average window.
Balanced (Default)	balanced	Balances cost and performance. Minimizes latency spikes while maintaining a bias toward lower-cost models.

// Set cost-arbitrage routing for a conversational assistant:
const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Summarize the log payload.' }],
  headers: {
    'x-selixes-routing-policy': 'cost-arbitrage',
    'x-selixes-latency-tolerance-ms': '2500' // Fail back to primary if arbitrage takes too long
  }
});

Expected Cost Savings

In typical production workloads, up to 70% of requests are simple instruction-following queries (e.g., formatting outputs, classifications, simple entity extractions) rather than deep reasoning tasks. By dynamically routing these to lower-cost models like gpt-4o-mini, claude-3-haiku, or a local llama-3-8b model, organizations typically see an average **30% reduction in total token expenses** while maintaining standard user experience latencies.