This article provides a step by step guide to serving Gemma 4 26B A4B, Google's quantization-aware trained (QAT) mixture-of-experts model, on an 8 ...
For further actions, you may consider blocking this person and/or reporting abuse
The drafter result is the most revealing number in this series: similar accepted-token behavior but only 1.24x instead of the 12B's 2.01x means speculative decoding here is bounded by expert fetch from RAM, not by draft quality. That makes the CPU-RAM link the real ceiling — have you tested how sensitive tok/s is to host memory bandwidth, e.g. DDR4-2133 vs 3200, to see whether faster RAM narrows the gap to the 12B? And on the cost side, 1.76x the GSM8K wall time for the same score makes the 26B hard to justify unless you genuinely need >81K context, which lines up with your conclusion that the 12B is the better trade-off.
I am running on ancient hardware- will see what I can eek out of my ram timings but ended up running DDR4 and the slower CPUs because thats what I had available.
Honestly, the ancient-hardware setup makes your numbers more useful, not less — most people running a 26B at home are on exactly that kind of box, not a tuned rig. If you do poke at the RAM timings, one thing worth isolating: whether tok/s moves with bandwidth or with CPU dispatch latency. Capping expert prefetch, or pinning the experts to fewer cores, would separate the two — if tok/s barely moves, it's the RAM link; if it jumps, the bottleneck was dispatch all along. Either answer sharpens the 12B-vs-26B trade-off you already landed on.
This is probably the low-end as the CPU RAM is DDR4 and the timings are not optimized. The CPU driving this box is a 3rd gen - but I suspect it is the DDR bandwidth that is the biggest driver
Great series entry. The result I found most interesting is that the 26B matches the 12B on GSM8K and BFCL while the 12B's drafter makes it 1.35x to 1.64x faster. That makes the "so which one?" section genuinely useful: the 12B up to ~82K tokens, the 26B only when you need the longer context. The MoE tip is also a good one, since active parameters (not file size) decide the speed once experts spill into RAM.
I appreciated that you labeled the explanation for the smaller drafter gain (1.24x vs 2.01x) as inferred rather than measured. It's a plausible story about each drafted token routing to its own experts, and I'd be curious whether a quick profile of CPU memory bandwidth during verification could confirm it. Since the CPU and RAM are in the serving path here, I'd also wonder how much the i7-5820K's quad-channel DDR4 shapes these numbers, and whether a newer DDR5 host would shift the 12B vs 26B tradeoff noticeably.
Thanks for including the thread-count tip and the full run times. Looking forward to the 31B results.
so far 12B looks like the best trade-off between quality and speed. but running more tests!
We run a 27B dense GGUF (Q5_K_XL) through llama.cpp for agent automation, so the "every token still crosses the CPU" part lands hard: with the experts parked in RAM the whole serving path becomes a memory-bandwidth problem, and that is why the held-context number (22.8 tok/s at 65k) looks like it walls on the DDR4 side rather than the GPU. The 131,072-token context on an 8 GB card is the part that matters most for our use case: long scraping/automation sessions where re-summarizing the conversation every few turns eats more time than the generation itself. Would be interesting to see what -ncmoe does when the RAM side is dual-channel instead of quad.
once it hits the slower CPU RAM that becomes the bottleneck. and good point on the memory timings. I am using a junk motherboard and generic DDR4 to recycle what I had but more aggresive memory timings might help.
which llama.cpp build are you serving this on, and is -ncmoe stable across releases? i've been tuning my own 26b-ish moe by moving whole layers between the card and the cpu, and the expert-only split changes the whole speed curve — 24 tok/s with everything in ram, then about half a tok/s per layer moved to the card. 131k context with the drafter on an 8gb card is the headline number here. one thing i'd watch: the cpu is in the serving path, so a backup job or a second model loading and the tok/s drifts for an hour. did the prompt-read side (302 → 227) stay flat across repeated runs, or does it get steeper once the cache fills up?
Built on llama.cpp 7182ef71e (2026-10-16). --n-cpu-moe has been in since August 2025, but I've only measured this curve on that one commit, so I wouldn't assume it holds across releases. Re-run the sweep after you update.
Prompt reading: the drafter and no-drafter runs each started their own server and read the same text separately. They agree within 1% at every depth (302/304 at 8k, 227/228 at 98k). It's a steady decline: 302, 288, 277, 253, 240, 227 at 8k to 98k. The prompt cache was on, so each figure is the speed of reading new text with the earlier text already held. It doesn't steepen near the top: the last 32k cost about 0.4 tok/s per thousand tokens, against 1–1.4 between 16k and 65k.
Contention: you're right, and I haven't tested it. All of these numbers are from an idle host, and decode reads every token's experts on the CPU. I'd expect a backup job to show up in tok/s, but that's a guess until it's measured.
All of the test data is in the repo.