Zero-Dropped-Token Stream Splicing During Upstream 504 Gateway Timeouts
The Streaming Dilemma in Production AI
When an application streams responses to an end-user or agent worker via Server-Sent Events (SSE), standard reverse proxies treat the stream as an all-or-nothing pipeline. If the upstream frontier provider (e.g., OpenAI) encounters an internal infrastructure stall, silent TCP drop, or 504 Gateway Timeout midway through a response, traditional load balancers terminate the connection abruptly.
The downstream consequences are disastrous:
- The client UI displays a broken half-sentence error block.
- Autonomous agent loops mistake the broken connection for a completed message and attempt to parse truncated JSON, generating invalid tool executions.
- All upstream tokens generated up to the failure point are billed, but completely wasted.
Socket-Level Stream Splicing Architecture
Selixes solves this with Zero-Dropped-Token Stream Splicing. Rather than acting as a dumb byte-pipe proxy, the gateway maintains an active buffer of delivered delta chunks at the socket boundary.
// Stream Splicing Failover Flow
1. Client connects via SSE -> Selixes Gateway
2. Gateway streams tokens from Primary Provider (OpenAI) -> Client UI
3. Primary Provider experiences TCP freeze / 504 timeout after 142 tokens
4. Selixes Socket Watchdog detects stalled heartbeat (>1,800ms silence)
5. Gateway pauses downstream socket flush without severing the TCP pipe
6. Failover Worker dispatches continuation prompt + prefix tokens to Standby (Claude/Gemini)
7. Standby stream is seamlessly spliced into the existing client socket
8. Client receives complete generation with zero connection resets
Why Single Base URL Architecture Wins
Custom client-side retry libraries require refactoring your entire codebase and re-implementing streaming handlers across every service. With Selixes, this resiliency is enforced at the network proxy layer. You swap two lines in your standard OpenAI SDK (baseURL and apiKey), and every downstream agent inherits autonomic failover, prompt caching, and zero-dropped-token stream splicing out of the box.
See It in Action
Selixes implements everything described in this article — circuit breaking, session budgets, local edge fallback, and private VPC deployment.