mini-vllm-rs is a single-node LLM inference engine I built in Rust on top of Candle for macOS. It can run CPU and Metal GPU backends concurrently, routing whole requests between them. My first request routing benchmark favored sending almost everything to GPU: the fastest policies assigned it 11 of 12 requests.
That left a more useful question unanswered: could selective CPU offload improve completion time compared with serving the entire workload on GPU alone? Separating prefill from decode suggested where to look: requests with warm prefixes, arriving behind a growing GPU backlog.
A follow-up experiment tested that idea. Routing 1 of 8 requests to CPU reduced the burst's completion time by 9.8% compared with forcing all 8 onto GPU. With 48 requests, offloading 6 to CPU increased aggregate output throughput by 12.4%.
The phase gap suggested the next experiment
The experiments used a quantized Qwen2.5 0.5B model with F32 activations on an Apple M2. For a 512-input/128-output request, isolated backend profiles showed:
| Phase | CPU | Metal GPU | GPU advantage |
|---|---|---|---|
| Prefill | 73 tok/s | 2,551 tok/s | About 35× |
| Decode | 45 tok/s | 293 tok/s | About 6.5× |
A long, uncached prompt exposed the CPU's largest disadvantage. Even when GPU already had work, avoiding CPU prefill could justify waiting for it. A warm prefix could remove most of that cost, leaving the smaller decode disadvantage for GPU backlog to outweigh.
Each backend owns its own scheduler and KV cache, and a request stays on the backend selected for it. I seeded an approximately 512-token prefix into both caches. Each measured request used the same 516-token prompt and generated exactly 128 tokens. Warming both sides gave CPU a chance to avoid expensive prefill.
I compared forced-GPU routing against mixed routing, where the router chose between CPU and Metal using estimated pending and incoming work. Both cases used the same scheduler settings, with up to 4 active requests per backend. Requests arrived 50 ms apart without waiting for earlier responses. Each case started with a fresh server and seeded caches, and ran 3 times.
What changed between requests 7 and 8?
The router priced prefill and decode separately:
score = remaining prefill tokens / prefill rate
+ remaining decode-token budget / decode rate
The totals included existing work plus the incoming request. Decode work was approximated using each existing request's unused max_new_tokens budget and the incoming request's full budget, so the router did not predict actual output lengths or early stopping. The rates came from isolated backend profiles, making this a work-based ranking rather than an exact prediction of batched completion time.
The candidate diagnostic logs show the boundary in run 1:
| Incoming request | CPU active / queued | GPU active / queued | CPU score | GPU score | Selection |
|---|---|---|---|---|---|
| 7 | 0 / 0 | 4 / 2 | 3.118 s | 3.027 s | GPU |
| 8 | 0 / 0 | 4 / 3 | 3.118 s | 3.444 s | CPU |
Estimated routing costs from mixed-routing run 1. All 3 runs made the same choices, with slightly different GPU scores.
Both candidates advertised a 496-token reusable prefix for the incoming 516-token prompt. The idle CPU therefore had only 20 estimated prefill tokens and a 128-token decode budget to account for. Its score stayed at 20/73 + 128/45 = 3.118 seconds.
At request 7, GPU already had 4 active requests and 2 queued requests. Each queued prompt also carried a 496-token cache estimate, leaving 20 prefill tokens apiece. Including the incoming request, GPU's estimated prefill work was therefore 2 × 20 + 20 = 60 tokens. Its decode budget was 256 queued + 496 active + 128 incoming = 880 tokens. The score was 60/2551 + 880/293 = 3.027 seconds, still below CPU's. Occupied GPU slots alone were not enough to justify offload.
About 50 ms later, request 8 reached the router. Request 7 had joined the GPU queue, while the active requests’ published remaining decode budget had fallen by 8 tokens. Including the incoming request, the GPU now had 80 estimated prefill tokens and a combined decode budget of 384 queued + 488 active + 128 incoming = 1,000 tokens. Its score rose to 3.444 seconds, above the idle CPU’s unchanged 3.118 seconds, so the router chose CPU. The warm prefixes kept prefill work small, and the accumulated decode work tipped the decision.
While helping me organize the earlier benchmark data, AI surfaced an accounting bug: queued requests lost their cache discount, inflating GPU prefill work. I decided to fix it and rerun the benchmark. The first CPU assignment moved from request 7 to request 8, and the capacity results here use the corrected runs. Queued requests now retain their routing-time cache estimates, though actual reuse may decrease or increase as the cache changes before execution.
Output-length estimation remains unfinished. For comparison, SGLang's scheduler scales remaining token budgets with an adjustable new_token_ratio to estimate future KV-cache demand for running requests, a heuristic rather than an exact output-length prediction.
Did the decision help in practice?
Run 1 produced the following measurements for request 8 and the burst as a whole:
| Measured outcome | Forced-GPU control | Mixed routing |
|---|---|---|
| Request 8: time to first token | 5.561 s | 0.339 s |
| Request 8: end-to-end latency | 11.116 s | 3.681 s |
| Whole-burst completion time | 11.486 s | 10.361 s |
Whole-burst time starts at the first submission, while request latency starts at that request's submission.
The offloaded request finished well before the mixed burst ended. CPU could start useful work while GPU continued serving its assigned requests, and the entire burst also finished sooner. Across all 3 runs, mean completion time fell from 11.47 to 10.35 seconds, a 9.8% reduction.
Mixed routing assigned 0, 1, and 6 requests to CPU, respectively. Wall time excludes startup and prefix seeding.
The benefit continued under a longer GPU backlog. In a 48-request burst, CPU received just 6 requests in every run. Mean completion time fell from 68.43 to 60.88 seconds. With the same total output, this 11.0% reduction in completion time translated into 12.4% higher aggregate output throughput.
Cells show routing estimates, not measured latencies.
Latency gains were not uniform. In the 8-request case, median TTFT improved, but p95 TTFT increased by 3.5%. Under the longer 48-request backlog, both median and p95 TTFT improved, alongside median and p95 end-to-end latency. Selective offload helped capacity without guaranteeing a win for every latency statistic.
A GPU concurrency sweep remains the next baseline check, since these gains were measured against the fixed 4-active-request GPU configuration. Sustained arrivals and varied cache-hit rates would test how far the result carries.
What the experiments changed
The first benchmark suggested where to look: the CPU’s disadvantage was much smaller in decode than in prefill. The follow-up showed that warm prefixes and GPU backlog could make selective CPU offload useful. Backend speed alone did not determine placement: the router also needed to account for the work already waiting and the prefill that cache reuse could avoid. For a broader discussion of the tradeoffs behind heterogeneous execution, see When Does Heterogeneous Inference Pay Off?.
The benchmark report contains the complete aggregate measurements and reproduction details.
Top comments (4)
Separating the scoring into prefill-rate vs decode-rate is the detail that makes this hold up — a lot of CPU-offload experiments fail because they treat a request as one uniform blob and the 35x prefill gap swamps everything. Routing on remaining prefill after a warm prefix is the right lever. One thing I'd love to see: how stable was the 9.8%/12.4% once request arrival got bursty and irregular rather than a fixed 50ms cadence? In our experience the win from CPU offload is real but fragile — it shows up precisely when the GPU backlog is deep, and a scheduler tuned on steady arrivals can route backwards under a spiky load and erase the gain. Also curious whether decode on CPU ever caused tail-latency complaints at 45 tok/s, since that's below most "feels responsive" thresholds even if aggregate throughput improves. Great to see someone actually measure this instead of assuming GPU-always.
Thanks for the thoughtful comment! You picked out exactly the mechanism I wanted to explore: warm prefixes reduce the CPU’s prefill disadvantage enough for the GPU backlog to matter.
One clarification: the router scores each incoming request using the current estimated remaining work on each backend, and it isn’t tuned to the 50 ms arrival interval. But that doesn’t establish robustness under irregular arrivals. This benchmark uses controlled bursts, and I haven’t tested how well the gains survive spiky traffic.
Your latency question is fair too. There’s no production deployment or user feedback here, and aggregate throughput alone doesn’t establish interactive responsiveness. Starting sooner on CPU but decoding more slowly creates a tradeoff worth evaluating further through TTFT and inter-token latency.
I wonder whether shared-prefix workloads on a constrained local device, or latency-tolerant background generation alongside interactive traffic, might offer useful real-world cases to investigate. Do those resemble the workloads where you’ve seen offload help?
My broader goal is to explore heterogeneous inference systems, which I also discussed in another post. CPU plus GPU gives me an accessible way to study these routing decisions, even if the gains from CPU offload are fragile. With another accelerator whose performance falls between the two, or whose prefill/decode balance differs, the useful operating window might be wider.
The separate prefill and decode scoring is the part I would carry into other heterogeneous setups, since a warm prefix makes the CPU's prefill penalty almost disappear and leaves queued decode work as the real bottleneck. The p95 TTFT increase in the 8-request case is worth digging into, though: did the offloaded request itself improve while a later GPU-queued request paid the price, or was the tail driven by CPU decode? If the router also tracked recent realized output lengths instead of only the full max_new_tokens budget, would the threshold between requests 7 and 8 move earlier or later under longer outputs?
Thanks! Separating prefill and decode costs is exactly the part I hope carries over to other heterogeneous setups.
Good question about p95 TTFT. Request 8 was the last arrival in that case, so there wasn’t a later incoming request, though requests were already queued on GPU. The aggregate p95 doesn’t tell us which request drove the increase. Identifying that requires a per-request breakdown. Slow CPU decode alone wouldn’t directly explain its own TTFT, since TTFT ends at the first token.
Using recent realized output lengths would be an interesting improvement, especially if conditioned on workload type. I wouldn’t assume a fixed direction for the crossover, though: shorter predictions reduce both the GPU’s estimated backlog and the CPU’s estimated cost of serving the incoming request. Longer predictions increase both, and CPU decode is substantially slower. Which effect dominates depends on the backlog and the relative rates.