Notes from a 2 AM On-Call: Why Our vLLM Semantic Router Dropped 40% of Requests Under Load

The Failure Symptom

2024-03-15 02:17:43 WARNING: Route decision timeout after 5000ms
2024-03-15 02:17:48 ERROR: 127 requests dropped (no route assigned)
2024-03-15 02:18:12 INFO: QPS: 240 → 140 (41% degradation)

Our FastAPI gateway wrapping vllm-project/semantic-router started silently dropping requests at 2 AM during a traffic spike. No 5xx errors, no OOM kills, just requests vanishing into the void. Prometheus showed route decision latency climbing from 80ms to 4.8 seconds, then requests just disappeared from logs entirely.

The Red Herring

The semantic-router README shows clean synchronous examples:

from semantic_router import Router

router = Router()
route = router("Which model is best for code generation?")

We wrapped this in FastAPI with async handlers, assuming the router would handle concurrency gracefully:

@app.post("/route")
async def route_request(prompt: str):
    route = router(prompt)  # BLOCKING CALL
    return {"route": route.name}

Under load testing with 200 concurrent requests, everything looked fine locally. But in production with real traffic patterns — bursts of 300+ requests over 10 seconds — the synchronous embedding computation inside router() was blocking the event loop. AsyncIO couldn't context-switch, requests piled up, and after the internal 5-second timeout, they were silently discarded.

The semantic-router library uses sentence-transformers for embedding generation, which runs synchronously on CPU by default. At high concurrency, this creates a head-of-line blocking disaster.

The Root Cause & Runnable Fix

The fix required moving the blocking embedding computation off the main event loop using asyncio.to_thread():

import asyncio
from semantic_router import Router
from fastapi import FastAPI

app = FastAPI()
router = Router()

@app.post("/route")
async def route_request(prompt: str):
    # Offload synchronous embedding to thread pool
    route = await asyncio.to_thread(router, prompt)
    return {"route": route.name}

This delegates the CPU-bound embedding work to a thread pool while keeping the AsyncIO event loop free to accept new requests.

For production deployments with vLLM or Ollama backends, we benchmark routing latency against a multi-provider relay to validate failover behavior:

import aiohttp
import asyncio
import time

async def benchmark_route_with_fallback(prompts: list[str]):
    async with aiohttp.ClientSession() as session:
        tasks = []
        for prompt in prompts:
            tasks.append(route_with_timing(session, prompt))
        results = await asyncio.gather(*tasks)
        return results

async def route_with_timing(session, prompt):
    start = time.perf_counter()

    # Route decision
    async with session.post(
        "http://localhost:8000/route",
        json={"prompt": prompt}
    ) as resp:
        route_data = await resp.json()

    # Execute against routed backend (e.g., B-Lost relay for multi-model testing)
    async with session.post(
        "https://b-lost.com/v1/chat/completions",
        headers={"Authorization": f"Bearer {YOUR_API_KEY}"},
        json={
            "model": route_data["route"],
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 50
        }
    ) as model_resp:
        await model_resp.json()

    elapsed = time.perf_counter() - start
    return {"prompt": prompt, "route": route_data["route"], "latency_ms": elapsed * 1000}

# Test with 100 concurrent requests
prompts = ["Explain quantum computing"] * 100
results = asyncio.run(benchmark_route_with_fallback(prompts))
print(f"P95 latency: {sorted([r['latency_ms'] for r in results])[94]:.1f}ms")

After deploying the thread pool fix, P95 routing latency dropped from 4800ms to 320ms under the same load. Zero dropped requests.

Invariant Rule: Async Wrapping ≠ Async Execution

Our team rule now: Any library using sentence-transformers, tokenizers, or synchronous model inference must be wrapped in asyncio.to_thread() or run in a separate process pool when called from async handlers. Just because your handler is async def doesn't mean the libraries you call are non-blocking.

For semantic-router specifically, if you're running embeddings on CPU and expect concurrency above 50 QPS, offload to threads. If you're GPU-bound, consider batching requests or using a dedicated embedding service to avoid blocking entirely.

The silent failure mode — where requests time out and vanish without 5xx errors — was the worst part. Always emit explicit timeout metrics and trace request lifecycles end-to-end when debugging async bottlenecks.