Before you wire up a multi-agent workflow, measure how fast your local models actually respond.
Model leaderboards rank quality, and spec sheets list parameter counts. Neither tells you how long a user, or the next agent in your chain, waits before the first token shows up. In an agent workflow, a router or triage step runs on every request. A few hundred milliseconds of extra latency there gets paid on every call.
So I measured time-to-first-token (TTFT) and end-to-end latency (E2E) for several small local models on a modest machine, before choosing any of them.
Test setup
- Hardware: M1 MacBook Air, 16GB unified memory, running on battery
- OS: macOS 27.2
-
Sampling:
temperature=0, 512 max output tokens - Method: the median of 5 formal runs per model
Models tested, all running locally:
| Model | Runtime | Notes |
|---|---|---|
Apple fm system
|
Local HTTP via fm serve
|
Apple's on-device system model |
| Google Gemma 4 E2B | MLX, 4-bit | MLX community revision 23876752
|
| Google Gemma 4 E4B | MLX, 4-bit | MLX community revision 475b9088
|
| Microsoft Phi-4 Mini | MLX, 4-bit | MLX community revision ac1c269c
|
| Mistral Ministral 3 3B | MLX, 4-bit | MLX community revision a962dcb0
|
The results
Medians of 5 formal runs each:
| Model | TTFT | E2E | Prompt tokens | Gen tokens | Peak RAM |
|---|---|---|---|---|---|
Apple fm system
|
0.693s | 0.886s | n/a | n/a | n/a |
| Gemma 4 E2B (4-bit) | 0.152s | 1.075s | 61 | 34 | 2.76 GB |
| Phi-4 Mini (4-bit) | 0.358s | 1.670s | 58 | 32 | 2.37 GB |
| Ministral 3 3B (empty system + regex fixed) | 0.564s | 1.764s | 72 | 34 | 2.14 GB |
| Gemma 4 E4B (4-bit) | 0.398s | 2.313s | 61 | 37 | 4.38 GB |
| Ministral 3 3B (default template, long system message) | 3.231s | 4.496s | 602 | 33 | n/a |
Two models lead, on different metrics:
- Gemma 4 E2B has the fastest TTFT at 152ms. The first token arrives almost immediately.
-
Apple
fm systemhas the fastest E2E at 886ms for these short, single-sentence outputs, even though its TTFT (0.693s) is much slower than Gemma E2B's.
Which model is fastest depends on which latency you care about.
TTFT and E2E answer different questions
TTFT is how long you wait before anything happens. It governs perceived responsiveness for interactive UIs and streaming. It also governs how quickly a classifier or router can start returning its decision.
E2E is the total time until the full response is complete. For short outputs, it tells you how long a blocking call takes.
Gemma E2B's profile is a very fast start with a slightly longer total (0.152s TTFT, 1.075s E2E). Apple FM starts slower but finishes the short answer sooner (0.693s TTFT, 0.886s E2E). If your agent needs to react the moment a request lands, TTFT matters most. If it is making blocking calls that return a short sentence, E2E matters more.
The two metrics are not interchangeable, so a benchmark that reports only one can mislead you.
The prompt template trap
The most useful finding was not about any model. It was about the prompt.
Ministral 3 3B appears twice in the table:
- With an empty system message and a fixed regex: 72 prompt tokens, 0.564s TTFT, 1.764s E2E
- With its default template and long system message: 602 prompt tokens, 3.231s TTFT, 4.496s E2E
The generated output was about the same size in both runs (34 vs 33 tokens). The model and hardware did not change. Only the template changed, and it added about 530 prompt tokens that had to be processed before the first token. TTFT went from 0.56s to 3.23s.
For fast agent routing, system prompt bloat is the real killer. Before you blame the model or the hardware, check how many tokens your default chat template injects. If you are running a short classification or tool-routing step, a lean prompt can matter more than the choice of model.
Non-thinking vs thinking modes
I also verified the token overhead at the template level for non-thinking and thinking modes. The reasoning is the same: any extra tokens the template adds ahead of your actual prompt show up as latency. Check the rendered prompt your runtime sends, not just the prompt you wrote.
Stop reason is not accuracy
A run that finishes with a clean stop reason has only told you that generation ended. It has not told you the answer was right.
When I read the actual outputs, I found edge cases that latency numbers would never show:
- Date boundary drift: dates near boundaries came back shifted.
- Missing owners: extracted action items sometimes dropped the person responsible.
A fast, wrong answer is still wrong. Treat latency as a filter for choosing candidate models, then check each candidate's outputs against your task before you commit to it.
Memory footprint on 16GB
The peak RAM numbers for the models that reported it:
- Ministral 3 3B: 2.14 GB
- Phi-4 Mini: 2.37 GB
- Gemma 4 E2B: 2.76 GB
- Gemma 4 E4B: 4.38 GB
On a 16GB machine, all of these fit. But your agent runtime, editor, browser, and other tools share the same unified memory. A small model that stays resident while you work is easier to live with than one that competes for memory. Gemma E4B costs roughly 1.6 GB more than E2B, and in this test it was slower on both TTFT (0.398s vs 0.152s) and E2E (2.313s vs 1.075s).
What this means for agent workflows
A common pattern for agent systems is to split the work:
- A small local model handles the fast, frequent decisions: intent classification, triage, tool routing, simple extraction.
- A larger cloud model handles heavy reasoning: planning, long-context analysis, and complex multi-step work, for example Gemini 2.5 Flash / Pro or Google Antigravity.
The measurements support this split. Gemma 4 E2B's 152ms TTFT is well suited to the first step. Routing happens on every request, so a sub-second start with a modest 2.76 GB footprint adds little overhead. Apple's fm system is a reasonable fit for quick one-sentence outputs, given its 886ms E2E. Neither is a replacement for a frontier model on hard reasoning, and I did not test them for it.
Architecting with Gemma & Gemini
For Google developers building agentic workflows, pairing on-device Gemma 4 E2B as a fast, zero-network dispatcher with cloud Gemini 2.5 Flash / Pro for complex reasoning gives the optimal balance between perceived latency and reasoning depth:
- Local Dispatcher (Gemma 4 E2B): Delivers instantaneous ~150ms TTFT to classify intents or parse local CLI parameters without incurring cloud round-trip latency (~300ms+) or API quota.
- Cloud Reasoner (Gemini): Takes over when tasks require deep context understanding, multi-turn tool calling, or complex synthesis.
Takeaways
- Measure TTFT and E2E separately. They rank the models differently, and they matter for different parts of a workflow.
- Audit your prompt template. A default system message took one model from 72 to 602 prompt tokens and its TTFT from 0.56s to 3.23s.
- Check outputs, not just stop reasons. Date drift and missing owners got past a clean finish.
- Match model size to the job. On this machine, the smaller Gemma E2B beat the larger E4B on latency and memory, so check whether you need the larger model at all.
- Measure on your own hardware. These numbers come from an M1 MacBook Air on battery, and your results will differ.
Run your own benchmark before you commit to a model for your agent stack. It costs little and it can change which model you pick.
Originally published at memo.jimmyliao.net.
Top comments (6)
The template result is the one I'd underline: same model, same machine, 72 vs 602 prompt tokens, and TTFT goes from 0.56 s to 3.23 s. The prompt is part of the system you're measuring.
A question on the method: how many warm-up calls ran before the five formal ones, and did the numbers drift if you kept calling for 20–30 minutes? On small edge boxes I've seen the first call and a long sustained run both land away from the median, and on battery the power policy can move things further. For a router that stays resident and gets called all day, that sustained number is the one I'd want next to the median.
Great question, and agree that the prompt template is part of the system being measured. I ran one warm-up call per configuration, followed by five measured invokes. I didn’t run a sustained 30 minute test, so can’t tell whether latency drifted over time or whether these medians represent an always-on router.
The run was also on battery, and I didn’t isolate power policy or thermal effects. The 72- versus 602-token configurations weren’t randomly interleaved, so I wouldn’t attribute the entire TTFT difference to prompt length alone.
A useful follow-up would measure repeated calls over time on both AC and battery, recording TTFT, end-to-end latency, memory pressure, and thermal state, but keeping the prompt and workload fixed.
Thanks for pointing out that gap.
Thanks, @jimmyliao — that's a rare kind of honesty about your own run, especially the point that the two templates weren't interleaved. If you do the AC vs battery follow-up, I'd read it closely; it's the same gap I keep running into on small boxes, where the median and the all-day number aren't the same thing.
For a router, the useful handoff may be the first valid, complete route rather than the first token: a fast partial label cannot safely dispatch work yet. Would you consider reporting time to a schema-valid decision alongside E2E latency and an abstain/error rate on the same labeled cases? That would make the latency comparison useful without treating a clean stop as a correct route.
For the router use case, the missing piece for me is what happens when the output has to be a valid route rather than just fast text. The date drift and missing owners you found already show that a clean stop reason is not enough, so a router would presumably need schema or enum validation, or grammar-constrained decoding, before it can hand off. Did you get a sense of whether that validation step eats into Gemma E2B's TTFT advantage? If the 152ms first token still needs a parse-and-retry loop before the route is trustworthy, the time to first valid route might rank the models differently from TTFT alone.
Report output tokens next to latency. On a local model the length of the answer often sets the wait more than the prompt does, so a latency table without output size says little.