Local LLM Speed by GPU Generation, and Our Production's Daily Token Volume

Model names, serving configs, and full prefill/decode measurements included

In my last post(new tab) I talked about switching to the RTX 5090. This time I'm going much deeper — exactly which model I run and how it's configured, precise prefill/decode speeds by hardware, and even the interim results of the model-adoption decision I'm currently running.

The hardware I've gone through

This bot's local LLM card has gone through one RTX 4070 Ti Super → two → one RX 7900 XTX → one RTX 5090.

I wrote up how I ended up actually buying the R9700 and putting it in (photos included) in a separate post(new tab) — it's actually a card I'd rejected as a main-card candidate multiple times before, bought again for a different purpose.

The actual model and serving config

This section is a snapshot as of when I wrote this post (08-27). Once the adoption decision below wrapped up, the model actually running in production — and the serving stack itself (vLLM → llama.cpp) — changed completely. See the update at the end for what's actually running now.

The model running in production every night is Qwen3.6-35B-A3B (MoE, 3B active), GGUF Q4_K_M quantization. Weights are 20.75GiB + 1.16GiB of KV/compute buffer = 21.91GiB in actual use (91% of the XTX's 24GB). The vLLM server runs on port 8200 with max-num-seqs 14, fp8 KV cache, prefix caching off, temperature 0.8, thinking mode off — this is the frozen production command.

At the same time, I'm evaluating whether to adopt Qwen3.8-27B (dense). Same conditions (fp8 KV cache, prefix caching off, temp 0.8, thinking off), but max-num-seqs is dropped to 3 — dense has different memory/compute characteristics than MoE, so concurrency has to be lower for stability.

MTP (speculative decoding) is always on. Acceptance rate is 50.1% for our MoE model, 66.95% for the 27B dense model in an external measurement, and anywhere from 81-97% in other external measurements — it varies a lot by model and settings.

Prefill/decode measurements — 27B dense

Same 27B dense model, roughly 12.7k prefill tokens per call, limited to cards I've actually run myself (at the time of writing I hadn't tested an AMD R9700 myself, so it only appears as a cited reference in the public-benchmark section below — for the numbers from actually buying and testing one, see the update at the end):

These two numbers are measured differently (single-stream vs. aggregate), so a straight "N times faster" division isn't quite fair. Still, looking at the 100-stock runtime alone, it dropped from 21.7 hours on the XTX to about 6.3 hours on the 5090 dense run. That said, pitting MoE against dense directly on the same 5090 tells a different story, covered below.

Raw decode speed by card — public cross-GPU benchmark

Decode speed by card for the same 27B-dense-class model with MTP on (source: arca.live cross-GPU thread(new tab); context depth uncontrolled, reference only):

The 5090 is about 1.13x faster than the XTX. I originally expected a much bigger gap based on the memory-bandwidth ratio (1.81x), but it turned out that conversion doesn't hold once MTP is on — speculative decoding frees decode from the bandwidth ceiling.

The R9700 row above (56 t/s) was someone else's measurement on a 27B-dense-class model. I later bought the card and measured it myself — not the same conditions, but this project's existing 35B MoE model, MTP on, single stream — and got 130-140 t/s decode. Not directly comparable to the table above since the model differs, but it was evidence the card itself isn't slow. See the update at the end for what it does paired with the 5090.

Decode speed from our own production logs, by hardware generation

ta_usage_tracker logs input/output tokens and elapsed time for every LLM call, per stock. I first averaged output_tokens ÷ elapsed_s per call, but vLLM batches multiple stock requests concurrently (continuous batching), so a single call's apparent speed swings with however many other requests happened to be in flight at that moment — that gave numbers that couldn't actually be used to compare hardware. So instead of averaging per-call rates, I recomputed it as total output tokens produced during that run ÷ the run's actual wall-clock duration (first call to last call).

For the 5090, instead of a "normal nightly batch" figure, here's exactly what actually produced today's ranking — this is the most accurate one.

Today's real ranking run — MoE vs dense on the 5090

I ran MoE and dense side by side for comparison, and the ranking that actually went out today used the dense results. Here's what it actually took each model to complete one full pass over 100 stocks (MoE ran yesterday evening, dense ran early this morning — each model's first complete pass):

Same 5090, same 100 stocks — MoE is about 7.7x faster than dense. Dense being slower isn't a hardware issue — it's inherent to the model architecture (MoE only computes its active parameters; dense computes all of them every time).

The model-adoption decision, in progress

Which model gets formally adopted isn't decided yet. Running today's ranking with dense for comparison doesn't mean it's formally adopted — one of the ongoing verification methods is self-QWK: running the same stock set multiple times and measuring how consistent the resulting grades are (quadratic weighted kappa). For MoE, I already have three repeated runs on the same 99 stocks (the MoE run in "today's real ranking run" above is one of these three):

The self-QWK band observed historically on the XTX was 0.08-0.21, so this value (0.1198) falls within it. The grade distribution skews toward "Hold" (71-80%), and QWK mechanically comes out lower when one grade dominates, so the raw agreement rate (56-61%) needs to be read alongside it.

Self-QWK for dense hasn't been measured yet (planned for this weekend). The metric that actually decides adoption isn't self-QWK — it's forward RankIC (the correlation between predicted and actual return ranking), and there are zero measurements of that so far. In other words, right now there's only a qualitative expectation that dense will perform better; whether it actually does is still unverified.

Daily token volume

This project has an AI pipeline that analyzes stocks with a local LLM(new tab).

Normal weekday average (MoE only, no side-by-side comparison running): about 5,200 calls, roughly 55 million input tokens, roughly 7.5 million output tokens.

Same single 100-stock pass as "Today's real ranking run" above:

Both models used the same call depth here (17 per stock), so the token volume itself is nearly identical — the difference shows up not in tokens, but in the wall-clock time above (46 min vs 6h 18m). For reference, on a normal day of MoE running solo, calls per stock can run much higher (55-71), since call depth itself depends on the purpose of that particular run (a formal production batch vs. a comparison run like this one).

What's next

Once forward RankIC is measured, I'll write up which model actually got adopted as a separate post.

Update (2026-09-08) - RTX 5090 + AMD R9700 mixed setup

The local LLM card lineup changed once more since this post. I added an R9700 alongside the 5090 (the story of how I ended up buying it, with build photos, is in this post(new tab)), splitting layers across both on a single llama.cpp server (-ts 23,25), and switched the model away from this post's MoE/dense candidates to Qwen3.8-Flash-Next (Q3_K_XL quantization). Here's the measured result from the first full 100-stock completion on this setup.

Adding a second card actually made the per-ticker time longer — that's not a hardware regression, it's that the model itself changed from something lighter than this post's MoE (3B active, 46-minute completion) to something heavier (6B active), while still being faster than this post's dense candidate. The full story — three quantization swaps and a full day of concurrency/batch-size tuning on this mixed-GPU setup — is written up in a separate post(new tab).