Size Your Model to Where You Run It: Gemma 4 E2B and E4B on an AWS L4 and Two Small GPUs

This article provides a step by step guide to serving Gemma 4 E2B and E4B, Google's quantization-aware trained (QAT) small models, on an NVIDIA L4 in an AWS EC2 G6 instance with llama.cpp, and compares the result with the same models on two small local GPUs: an 8 GB RTX 3050 desktop card and a 4 GB GTX 1650 Ti laptop card. Every card runs the same llama.cpp build, the same model files, the same drafter and the same tests. The measures are generation speed for one user and for many, what the drafter adds on each card and why, answer quality, and cost.

https://github.com/xbill9/gemma4-dev

For one user, the 24 GB L4 is 1.2-1.3x an 8 GB desktop card, with the same answers. The 4-bit small models use 13-29% of the L4's memory, and the rest of the card sits idle:

The desktop card delivers 79-81% of the L4 without the drafter and 69-77% with it. The L4 earns its memory with many users at once (1.6x the 3050 at 64) and with models too large for 8 GB. The rule this article arrives at: size the model to where it runs.

What Is Being Measured?

Gemma 4 E2B and E4B are the two small models of the family, served here as exact 4-bit (Q4_0) GGUF files whose every block sits on Google's QAT grid: xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf and xbill9/gemma-4-E4B-it-qat-q4_0-exact-gguf. Google also ships a QAT drafter for each, a small model that proposes the next few tokens for the main model to check in one pass; it is stored the same way, as xbill9/gemma-4-E{2,4}B-it-qat-assistant-q4_0-exact-gguf.

When one user is generating, every token reads every weight of the model once. The speed limit is then how fast the card reads its memory, and the size of the memory plays no part as long as the model fits. That is why a card with three times the memory can be only a little faster.

The three cards:

All three run llama.cpp 77c4540ce with flash attention, a 16-bit context cache and the per-layer embedding table on the card. The L4 build uses CUDA 13.1 for sm_89.

Step 1 — Serve the Rigs

Each model has a rig directory with a tpu.env holding every setting and a Makefile to run on the instance:

git clone https://github.com/xbill9/gemma4-dev
cd gemma4-dev/gpu-llamacpp-g6-2b-q4_0      # or gpu-llamacpp-g6-4b-q4_0
make toolkit build download draft serve

The G6 instance comes from gpu-pytorch-g6-2b-q4w4a16's MCP tool create_g6_instance, on the PyTorch Deep Learning AMI. The AMI has an NVIDIA driver (595.112.07, CUDA 13.2) and no CUDA compiler, so make toolkit installs CUDA 13.1 from NVIDIA's apt repository, one minor version below what the driver supports. make build compiles llama.cpp for sm_89 in 633 s.

The serving line, E2B:

llama-server -m gemma-4-E2B-it-q4_0-exact.gguf -ngl 99 -c 131072 -ctk f16 -ctv f16 -fa 1 \
  -t 4 -tb 8 --parallel 1 --reasoning off --metrics -ot 'per_layer_token_embd\.weight=CUDA0' \
  -md gemma-4-E2B-it-qat-assistant-q4_0-exact.gguf --spec-type draft-mtp -ngld all --spec-draft-n-max 2

At the full 131,072-token context with the drafter, E2B peaks at 3,952 MiB and E4B at 6,774 MiB of the L4's 23,034.

Step 2 — Measure One User

speed.py sends six prompts (code, chat, story, explanation, facts, summary), 256 tokens each, three greedy repeats per prompt with the prompt cache off, and reports the geometric mean over prompts. Each arm gets its own server, started on a cooled card. On the L4:

A no-drafter arm run last lands 0.7% (E2B) and 1.2% (E4B) under the first. Code gains most from the drafter (2.39x E2B, 2.55x E4B at 3 tokens) and stories least (1.31x, 1.38x).

With the drafter, E4B runs at 0.972x E2B without it: 129.7 against 133.4 tok/s.

The same prompts on the two local cards, best setting per card:

🔎 Tip: The Card Sets the Draft Length

The drafter's guesses are kept at the same rate on every card. What changes is what each guess costs. Splitting each reply into steps (one pass of the main model over the last token plus the drafted ones) shows it, here for E2B:

The speed-up is tokens per step divided by step time. On the L4 llama-bench times the main model over 1 to 6 tokens in one pass: 4 tokens take at most 1.04x the time of 1, at 0, 8,192 and 32,768 tokens of context. So checking 3 guesses is close to free there, and the added time is the drafter's own, 0.75-1.18 ms per guess. That is 33% of an E2B step at 3 guesses and 19% of E4B's longer step, which is why E4B gains more from its drafter. A slower card pays more per guess and peaks at fewer.

Step 3 — Hold a Long Conversation

longctx.py puts the first D tokens of a long document ahead of a story, a code and a summary request, at -c 131072. Mean speed-up over the requests at 2 drafted tokens:

Generation without the drafter slows with context (E2B story 134.1 tok/s empty, 66.7 at 120,000 ahead on the L4), and the drafter keeps paying at every depth on both cards.

Step 4 — Serve Many Users at Once

mixed.py sends c requests at once, each one of the six chat prompts, 256 tokens, five repeats, one shared context cache (-kvu) at -c 32768 --parallel 64. Total output tok/s:

This is where the L4's size shows: 1.61-1.65x the 3050 at 64 requests. The drafter (2 tokens) helps on the L4 up to 16 requests (E2B 1.06-1.19x, E4B 1.15-1.34x) and costs output at 64 (0.87x for both).

Step 5 — Check the Answers

gen_eval.py runs GSM8K (1,319 grade-school math questions, up to 2,048 tokens per reply) and BFCL simple (400 single function calls) at -c 32768, 8 requests at once, with and without the drafter at 2 tokens:

Paired question by question, every drafter-against-none difference has a 95% range that includes zero. The 3050 scored E2B 0.8984 / 0.9022 and E4B 0.9340 / 0.9325 on GSM8K under the same settings: the same model files give the same answers on every card.

E4B answers more GSM8K questions right than E2B, +0.0318 without the drafter (range +0.0167 to +0.0470) and +0.0303 with it, and BFCL is level between the two. With its drafter E4B finishes GSM8K in 893 s, sooner than E2B without one (1,014 s).

Step 6 — Price It

The L4 instance, g6.2xlarge in us-east-1, listed at $0.9776/h on demand and $0.8393/h spot on the day of the runs. Per million output tokens, one machine kept busy:

One user pays for the whole card while using 13-29% of its memory. Filled with 64 requests, the same card costs a sixth to a seventh as much per token. A local card's cost is the card and its electricity, neither measured here.

🔎 Tip: Where the Bigger Card Earns Its Keep

The 8 GB 3050 runs every larger Gemma 4 too, at a price paid in speed once the model outgrows the card. From the same series on the same card:

All three fit a 24 GB card whole; this article measures them on the 3050 only.

Compare and Contrast

So, Which One?

For one user on E2B or E4B, a local 8 GB card. It delivers 69-81% of the L4's speed with the same answers, and the model uses a fraction of either card.

For many users at once, the L4. At 64 requests it serves 1.6x the 3050's output, and at about $0.25-0.30 per million tokens on spot.

For quality at small-model speed, E4B with its drafter. On the L4 it generates at 0.97x E2B without a drafter and scores 3 points higher on GSM8K.

For the 26B and 31B, a card that holds them. On 8 GB they spill into system RAM and slow to 34 and 4 tok/s.

Summary

The goal of this article was to find what a 24 GB cloud GPU buys over small local GPUs when serving Gemma 4's small models. The key to the answer was running the same llama.cpp build, model files, drafter and tests on all three cards. The results were:

• 🟢 One user: the L4 is 1.23-1.27x the 3050 without the drafter and 1.30-1.44x with it, at 13-29% of its memory
• 🟢 The same model files give the same answers on every card; the drafter costs no GSM8K or BFCL answers
• 🟢 The drafter's guesses are kept at the same rate on every card; the card's cost per guess sets the best draft length, 1 / 2 / 3 tokens on the 1650 Ti / 3050 / L4
• 🟢 E4B with its drafter generates at 0.97x E2B without one, 3 GSM8K points higher
• 🟢 64 users: the L4 serves 1.6x the 3050, $0.25-0.30 per million tokens on spot
• ⚠️ The drafter helps on the L4 up to 16 users at once and costs output at 64
• ❌ One user on the L4 pays for a whole card and uses under a third of its memory

Scope: Gemma 4 E2B-it and E4B-it QAT only, exact Q4_0 GGUF files and exact Q4_0 QAT drafters, text only, llama.cpp 77c4540ce on all three cards, 16-bit context cache, thinking off. One L4 instance on one day (spot g6.2xlarge, us-east-1d); the 3050 (i7-5820K desktop) and 1650 Ti (laptop) runs are from 2026-10-14 and 2026-10-15 on different hosts with the same prompts and scripts, so cross-card ratios are indicative. The many-users test uses short chat prompts and a shared context cache. Larger cloud GPUs and the 12B, 26B and 31B on the L4 were not run.

The strategy for using MCP for local accelerator comparison was validated with an incremental step by step approach.

References

• Part 2, Do Drafters Work on Small Models?: https://dev.to/gde/do-drafters-work-on-small-models-gemma-4-e2b-and-e4b-on-two-small-gpus-1b4g
• Part 6, A Dense 31B on an 8 GB GPU: https://dev.to/gde/a-dense-31b-on-an-8-gb-gpu-gemma-4-31b-on-an-rtx-3050-26p9
• Repository: https://github.com/xbill9/gemma4-dev
• E2B on the L4: https://github.com/xbill9/gemma4-dev/blob/main/gpu-llamacpp-g6-2b-q4_0/benchmarks/runs/2026-10-26-drafter-g6/REPORT.md
• E4B on the L4: https://github.com/xbill9/gemma4-dev/blob/main/gpu-llamacpp-g6-4b-q4_0/benchmarks/runs/2026-10-26-drafter-g6/REPORT.md
• Drafter cost and many users on the L4: https://github.com/xbill9/gemma4-dev/blob/main/gpu-llamacpp-g6-2b-q4_0/benchmarks/runs/2026-10-26-drafter-cost-g6/REPORT.md
• E2B GGUF: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf
• E4B GGUF: https://huggingface.co/xbill9/gemma-4-E4B-it-qat-q4_0-exact-gguf
• E2B drafter: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-assistant-q4_0-exact-gguf
• E4B drafter: https://huggingface.co/xbill9/gemma-4-E4B-it-qat-assistant-q4_0-exact-gguf
• llama.cpp: https://github.com/ggml-org/llama.cpp
• Amazon EC2 G6 instances: https://aws.amazon.com/ec2/instance-types/g6/