DEV Community

Cover image for Notes from a 2 AM On-Call: Why Our Bifrost Gateway Silently Dropped 40% of Streaming Requests
James LIN
James LIN

Posted on

Notes from a 2 AM On-Call: Why Our Bifrost Gateway Silently Dropped 40% of Streaming Requests

#devops #cloud #api #productivity

Notes from a 2 AM On-Call: Why Our Bifrost Gateway Silently Dropped 40% of Streaming Requests

At 2:14 AM last Tuesday, our on-call dashboard lit up. The symptom was bizarre: SSE streaming responses from our maximhq/bifrost gateway were working perfectly for the first 5-10 concurrent requests, then silently failing for everything beyond that threshold. No 502s, no connection resets, no OOM kills in the logs. Just... silence. The client SDKs sat there waiting for chunks that never arrived.

The Failure Symptom

Our stack: maximhq/bifrost (v0.8.x) running behind nginx as a reverse proxy, routing to multiple LLM providers with streaming enabled. When concurrent streaming requests hit ~12-15 simultaneous connections, new clients would establish WebSocket handshakes successfully, receive the initial SSE headers, then get zero data chunks. The bifrost process showed no errors. Nginx access logs showed 200 responses with tiny byte counts.

The raw symptom in our monitoring:

SSE stream success rate: 100% (0-10 concurrent)
SSE stream success rate: 61% (10-15 concurrent)
SSE stream success rate: 43% (15+ concurrent)
nginx upstream response time: 0.001s (failing requests)
bifrost process CPU: 8%, memory: stable
Enter fullscreen mode Exit fullscreen mode

Clients were connecting, bifrost was accepting, but the streaming bodies vanished into a black hole somewhere in the proxy layer.

The Red Herring

Every standard nginx streaming tutorial suggests the same directives:

proxy_buffering off;
proxy_cache off;
proxy_http_version 1.1;
Enter fullscreen mode Exit fullscreen mode

We had all of these set correctly. The issue wasn't request buffering—it was response buffering at the OS socket level combined with nginx's default behavior for upstream keepalive connections.

Our initial hypothesis: bifrost's streaming implementation had a concurrency bug. We spent an hour tracing through the Go source, added debug logging to every SSE write call, and confirmed bifrost was writing chunks correctly to all client sockets. The data was leaving the process. It just wasn't arriving at clients.

The Root Cause & Runnable Fix

The actual culprit: nginx's proxy_buffering off only disables request body buffering to the upstream. For SSE/streaming responses, you also need to explicitly flush response buffers and prevent nginx from waiting to accumulate data before sending it downstream. Under concurrent load, nginx was holding response chunks in per-connection buffers waiting for more data or a buffer size threshold, effectively stalling the stream.

The fix required three additional directives in our nginx location block:

location /v1/ {
    proxy_pass http://bifrost_upstream;
    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_cache off;

    # The actual fix:
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
    chunked_transfer_encoding on;  # Force immediate chunk relay

    # Critical for SSE:
    proxy_set_header Connection '';
    proxy_set_header X-Accel-Buffering no;  # Disable all response buffering
}
Enter fullscreen mode Exit fullscreen mode

The X-Accel-Buffering: no header is the kill switch. It tells nginx to immediately relay every chunk from the upstream without accumulating or waiting. Combined with chunked_transfer_encoding on, this forces nginx to respect the streaming contract.

After deploying this config:

SSE stream success rate: 100% (0-50 concurrent)
avg TTFT: 340ms (unchanged)
no increase in nginx CPU or connection overhead
Enter fullscreen mode Exit fullscreen mode

The Production Context

Our team routes requests through bifrost to multiple providers (OpenAI, Anthropic, and a relay through https://b-lost.com/v1 for fallback and quota management). When we benchmark multi-provider streaming through this topology, the nginx buffering issue only manifested under real concurrent load—single-threaded curl tests always succeeded because they never triggered the buffer accumulation threshold.

The insidious part: bifrost's own metrics showed successful writes. Nginx logs showed 200s. Clients saw established connections. But the streaming payload delivery silently stalled due to a proxy buffering behavior that only surfaces under concurrent SSE streams.

Invariant Rule: Streaming Proxy Verification Protocol

Every proxy layer in front of a streaming gateway must be load-tested with concurrent streaming requests, not just sequential curl loops. Our new pre-deploy verification:

# Generate 20 concurrent streaming requests
seq 1 20 | xargs -P 20 -I {} curl -N -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"Count to 50"}],"stream":true}' \
  https://gateway.internal/v1/chat/completions &

# All 20 must deliver chunks within 2s
# Any silent stall = failed verification
Enter fullscreen mode Exit fullscreen mode

If any request establishes a connection but delivers zero chunks within 2 seconds, the proxy config is broken for production streaming workloads. This caught the nginx buffering issue in our staging environment before it could reoccur.

The 3-line fix (X-Accel-Buffering no, chunked_transfer_encoding on, explicit timeout extensions) is now a mandatory template in every nginx location block that proxies to bifrost or any streaming AI gateway.

Top comments (0)