DEV Community

Pingredsai
Pingredsai

Posted on

Why Your Phone Runs LLMs 80x Slower Than It Should (And What I Found)

#android #llm #performance #kotlin

Why Your Phone Runs LLMs 80x Slower Than It Should (A Debugging Log)

Tags: on-device inference / llama.cpp / Android / performance
Status: draft

I ran Qwen2.5-1.5B-Instruct Q4_K_M fully offline on a Google Pixel 4 (Snapdragon 855, 2019).

Metric Measured
Generation 0.5 tok/s
First token 1.9 s
Peak RSS 1.3-2.0 GB
CPU 402% (4 threads saturated)

Theoretical limits say this device should do 30-60 tok/s.
It is 60-120x slower. And the CPU is already maxed out.

Here is the full debugging log — including three mistakes I made.

1. Do the math first

Memory bandwidth bound:

model size      1.06 GB
SD855 bandwidth ~34 GB/s
one token reads all weights once
-> theoretical ceiling ~32 tok/s
Enter fullscreen mode Exit fullscreen mode

Compute bound:

1.5B params x 2 FLOPs = 3 GFLOPs per token
SD855 CPU peak ~182 GFLOPS
-> theoretical ceiling ~60 tok/s
Enter fullscreen mode Exit fullscreen mode

Both ceilings land in the 30-60 tok/s range. Measured: 0.5.

Knowing the gap (60-120x) tells you what to suspect.

2. Suspect 1: thread count? (ruled out) 

adb shell top -b -n 1 -o PID,%CPU,%MEM,ARGS
Enter fullscreen mode Exit fullscreen mode 
PID    %CPU   %MEM   ARGS
17834  402%   24.3%  com.example.localai
Enter fullscreen mode Exit fullscreen mode

402% — four threads fully saturated. Not a threading problem. Ruled out.

3. Suspect 2: missing ARM optimization? (HIT)

This is the valuable one.

llama.cpp does its heavy lifting in ggml. From ggml/src/ggml-cpu/CMakeLists.txt:

if (GGML_NATIVE)
    # probe host CPU, add -mcpu=native
    ...
else()
    if (GGML_CPU_ARM_ARCH)
        list(APPEND ARCH_FLAGS -march=${GGML_CPU_ARM_ARCH})   # <-- key
    elseif(GGML_CPU_ALL_VARIANTS)
        ...
    # neither set -> NO -march at all
endif()
Enter fullscreen mode Exit fullscreen mode

When cross-compiling for Android:

  • GGML_NATIVE is OFF (can't probe the target)
  • GGML_CPU_ARM_ARCH defaults to empty
  • Result: no -march is passed at all

Compiler falls back to the baseline ARMv8-A — no dotprod, no fp16.

Quantized matrix multiplication (99% of LLM compute) degrades to a slow path.

Fix:

arguments += listOf(
    "-DUSE_LLAMA_CPP=ON",
    "-DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16"
)
Enter fullscreen mode Exit fullscreen mode

Result: prefill went from 114 s to 45 s — 2.5x faster.

Better. But not enough.

Verify the optimization actually landed

Don't trust the flag. Disassemble:

llvm-objdump -d libggml-cpu.so | grep sdot
Enter fullscreen mode Exit fullscreen mode 
18f558: 4e829420    sdot   v0.4s, v1.16b, v2.16b
Enter fullscreen mode Exit fullscreen mode

sdot is there. dotprod is active. Ruled out.

Still 0.5 tok/s.

4. Suspect 3: mmap pages getting evicted? (ruled out)

Android reclaims mmap'ed pages under memory pressure. Then every weight read falls back to flash:

flash read ~1.5 GB/s
1.06 GB / 1.5 GB/s ~ 700 ms/token
Enter fullscreen mode Exit fullscreen mode

Right order of magnitude for the 2 s/token we saw.

So I forced the model to stay resident:

mparams.load_mode = LLAMA_LOAD_MODE_MLOCK;
Enter fullscreen mode Exit fullscreen mode

Result: still 0.5 tok/s. Ruled out.

5. Three mistakes I made (worth avoiding)

Mistake 1: assuming llama_batch_get_one pins position to 0

I claimed: "llama_batch_get_one() sets pos to 0, so every token recomputes the whole sequence."

Then I hand-rolled llama_batch_init() with explicit positions.

What actually happened:

  1. Reading the header: llama_batch_get_one's pos is nullptr — llama.cpp assigns positions automatically. Not 0.
  2. My hand-rolled version hung the prefill (43 tokens, never returned).
  3. Reverting to llama_batch_get_one fixed it.

Lesson: for perf issues, suspect compiler flags and memory access first — not a detail you "remember".

Mistake 2: using a removed API 

llama_kv_cache_clear(g_ctx);   // removed in current llama.cpp
Enter fullscreen mode Exit fullscreen mode

Now:

llama_memory_clear(llama_get_memory(g_ctx), true);
Enter fullscreen mode Exit fullscreen mode

Mistake 3: guessing struct field names 

mparams.use_mmap  = false;   // fields no longer exist
mparams.use_mlock = true;
Enter fullscreen mode Exit fullscreen mode

Merged into one enum:

mparams.load_mode = LLAMA_LOAD_MODE_MLOCK;
Enter fullscreen mode Exit fullscreen mode

Common lesson: llama.cpp's API moves fast. Always read include/llama.h. Never write from memory.

6. Where it landed

After ruling out three suspects:

3 GFLOPs per token
2 s per token measured
-> ~0.8% of peak CPU utilization
Enter fullscreen mode Exit fullscreen mode

And it is not a bandwidth wall (~500 MB/s measured vs ~34 GB/s available).

So compute efficiency itself is the problem — beyond "wrong config". Could be Android CPU scheduling, OpenMP synchronization overhead, or platform-specific behavior in llama.cpp.

That is not a one-day fix. And it should not block shipping.

7. What I did about it

  1. Ship it and state the number honestly — README says "Pixel 4: 0.5 tok/s"
  2. Explain the limitation — old SoC + large model
  3. Ask the community for more data — Snapdragon 8 Gen 2/3 results welcome
  4. Move performance work to v2 instead of blocking v1

8. Transferable lessons

Lesson Why
Check compiler flags first -march / dotprod issues are the #1 cause of slow on-device inference
CPU saturated ≠ efficient 402% usage can mean you're running the slowest implementation
Compute theoretical limits The gap tells you what to suspect
Verify with disassembly Flags set ≠ instructions generated
Read the header, don't recall it llama.cpp API churns fast
Set a stop-loss for the project Perf work must not block shipping forever

Appendix: environment

Item Value
Device Google Pixel 4 (Snapdragon 855, 6GB, Android 13)
Model Qwen2.5-1.5B-Instruct-Q4_K_M.gguf (1065 MB)
llama.cpp commit 8a1a9b5 (2026-10-09)
NDK 30.0.16248370
CMake 4.1.2
Build flags -DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16, -DUSE_LLAMA_CPP=ON
Runtime n_threads=4, n_ctx=2048, load_mode=MLOCK

Source code

Full implementation (Kotlin + JNI + llama.cpp + Jetpack Compose), MIT licensed:

https://github.com/Pingredsai/local-ai-android

Running the same model on newer hardware? Send me your numbers — that is exactly what this project needs.

Top comments (5)

Collapse
 
alfred_odong_322108a5cc3d profile image
Alfred Odong •

Great debugging log, and the "do the math first" step is the part most people skip. Two related traps from the desktop side that look identical to a slow model and aren't:

  1. Scheduling, not compute. A llama.cpp server I started from a tool that runs at nice 5 got its child processes parked on the efficiency cores under load, and a 3-second transcription took 37 seconds. Same binary, same flags; the only difference was who launched it.

  2. Storage, not compute. Running off a USB stick, the whole model is read at every start, so on a 34 MB/s drive a 2.6 GB model took over a minute before the first token, and seconds on a USB 3 drive. Tokens per second was fine both times; the stopwatch said otherwise.

Your -march finding is the same shape: the number that looks like "the model is slow" is almost never the model.

Collapse
 
pingredsai profile image
Pingredsai •

Thanks, this is a great pair of examples — both hit close to home.

The scheduling one is especially relevant. Android's big.LITTLE scheduler is
aggressive about migrating long-running threads, and I measured 402% CPU
(4 threads busy) but never checked which cores they actually landed on.
That's my next test — reading per-thread affinity from /proc, or pinning
threads with sched_setaffinity and comparing.

The storage one matches an experiment I ran: I switched to
LLAMA_LOAD_MODE_MLOCK to force the model resident instead of mmap'd,
expecting a big win. It made no difference (0.5 tok/s either way), which told
me the pages weren't being evicted. But your point makes me want to measure
the steady-state read rate separately from startup cost.

Agreed on the shape of the bug: the number that looks like "the model is slow"
is almost never the model. I'll report back on the affinity test.

Code: github.com/Pingredsai/local-ai-and...

Collapse
 
pingredsai profile image
Pingredsai •

Thanks — I went and tested both of your hypotheses. Results:

Scheduling: not it (on this device).

I read field 39 (current CPU) of /proc//task//stat while inference
was running. All four inference threads stayed on cpu4-7 (the A76 mid/big
cores) and never touched cpu0-3 (A55). OpenMP is working too — three
openmp_worker threads plus the main thread, matching n_threads=4.

Frequency management: real, but small.

I did find heavy throttling though. Under schedutil, cpu4/cpu6 oscillate
between 710 and 1920 MHz (max 2419), and cpu7 never exceeded 2016 (max 2841).
Battery at 33.9 °C, so it's not thermal — it's schedutil responding badly to
bursty load.

So I rooted the device, chmod'd the sysfs nodes, and pinned all 8 cores to
performance (1785 / 2419 / 2841 MHz):

schedutil 0.5 tok/s first token 1863 ms
performance 0.6 tok/s first token 1619 ms

Only +20%. Real, but not the dominant factor.

Where that leaves me:

CPU at full clock, threads on big cores, dotprod instructions confirmed present
— and utilization is still ~1% (3 GFLOPs / 1.67 s ~= 1.8 GFLOPS against a
~182 GFLOPS peak). Memory bandwidth isn't the wall either (~635 MB/s vs
17-34 GB/s available).

So the bottleneck looks like it sits inside ggml's quantized kernels on this
platform — memory access patterns, cache behaviour, or OpenMP barrier overhead
on short work units.

Your storage point I couldn't reproduce: switching to LLAMA_LOAD_MODE_MLOCK
made no difference, so pages weren't being evicted in my case.

Appreciate the pointers — the scheduling check alone was worth doing, even
coming back negative. It's now a bounded question instead of a vague "it's slow".

Code: github.com/Pingredsai/local-ai-and...

Collapse
 
alfred_odong_322108a5cc3d profile image
Alfred Odong •

Thanks for running both and writing up the negative results; that is rarer than it should be. With scheduling, frequency, dotprod and bandwidth all ruled out and utilisation at 1%, I would stop looking at the hardware and bisect the software in two cuts.

  1. Is it ggml or is it the app? Build llama-bench with the exact same CMake flags, adb push it, and run it on the same GGUF: ./llama-bench -m qwen2.5-1.5b-q4_k_m.gguf -t 4 -p 64 -n 32. If llama-bench does 15-30 tok/s, the kernels are fine and the 0.5 lives in the JNI/app layer. If llama-bench is also at 0.5, it is the build.

  2. Build type. A Gradle debug variant hands CMake CMAKE_BUILD_TYPE=Debug unless you override it, and ggml at -O0 with asserts on is 20-50x slower than release. sdot showing up in the disassembly does not tell you the optimisation level. Add -DCMAKE_BUILD_TYPE=Release to that arguments list, or benchmark the release variant.

If it lands in the app layer, the two things I have seen produce exactly "threads busy, 1% useful work": a decode loop that clears or recreates the KV cache every token, so each token is a full prefill (watch n_past; 2 s/token is suspiciously close to your prefill cost per token), and OpenMP spin barriers across the 3+1 A76 clusters. For the second, -DGGML_OPENMP=OFF to use ggml's own thread pool and n_threads=3 so every thread sits on the same mid cores is a cheap test.

Curious which cut it lands in. I would bet the build type.

Thread Thread
 
pingredsai profile image
Pingredsai •

You called it — it was the build type.

I checked the previous build.ninja: zero -O flags anywhere. Not -O0
explicitly, just none. CMake's Debug config with no optimisation. And the
project was building the debug variant, so AGP was handing
CMAKE_BUILD_TYPE=Debug to CMake.

Fix: force -O3 in CMakeLists (before add_subdirectory(llama.cpp)) plus
-DCMAKE_BUILD_TYPE=Release in the Gradle arguments.

Results on the same Pixel 4, same GGUF, same model:

before   prefill 43 tok = 42 s      0.5 tok/s    first token 1863 ms
after    prefill 43 tok = 0.556 s   18.5 tok/s   first token 52 ms
Enter fullscreen mode Exit fullscreen mode

37x on generation, 75x on prefill.

Also: the .so sizes were a useful confirmation. libggml-cpu.so went from
3.92 MB to 0.95 MB, which is a decent smell test for "did optimisation
actually happen".

Your point about sdot in the disassembly not telling you the optimisation
level was exactly right — that's what sent me down the wrong path for days.
I never checked the most basic thing.

llama-bench is still on my list as the next cut, but this one landed first.

Thanks — genuinely. This saved me a lot of time.