DEV Community

Cover image for Same Local Model, 3 Coding Agents: pi 12/12, Codex 12/12, Claude Code 7/12 on a 6GB GPU
KUN Field Notes
KUN Field Notes

Posted on Originally published at kunbuilds.com AI-assisted

Same Local Model, 3 Coding Agents: pi 12/12, Codex 12/12, Claude Code 7/12 on a 6GB GPU

#localllm #llamacpp #ai #claudecode

Claude Code told me the web page was done. The folder was empty.

That was the worst moment of a small test: one local model on a 6GB laptop GPU, plugged into three coding agents. Same model, same tasks. Only the agent changed.

Setup

  • RTX 3060 Laptop 6GB + 64GB RAM, WSL2 Ubuntu 24.04
  • llama.cpp 9655061: llama-server -c 32768 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 --jinja; experts on CPU (-ncmoe 36 for LFM, -ncmoe 40 for Qwen). Fully offline.
  • Models: LFM2.5-24B-A2B Q4_0 (3 runs per task), Qwen3.6-35B-A3B UD-Q4_K_XL (1 run, control)
  • Tasks:
    • A: fix two bugs in calc.py until pytest passes
    • B: single-file timer page, checked by a script against 8 items
    • Pelican: Generate an SVG of a pelican riding a bicycle
  • 900 s cap per run

Results

Agent Passed LFM bug-fix median First prompt
pi 12/12 47 s ~1,400 tokens
Codex 12/12 85 s ~6,400 tokens
Claude Code 7/12 132 s ~15,800 tokens

Tasks passed

The prompt is the problem

Every request starts with the agent's system prompt and tool definitions. First prompt eval in the llama-server log:

Agent Tokens Time
pi 1,442 5,666 ms
Codex 6,429 18,567 ms
Claude Code ~15,800 48,837 ms

With a 32K context, Claude Code fills about half of it before doing anything, and reading that prompt takes about 8.6× longer than pi's.

pi

12/12 on both models, zero tool errors in the logs, fastest bug fix (47 s median). I also trimmed its prompt to about 750 tokens with only read/edit/bash enabled. Still 12/12, same speed. Harder tasks might separate the two; these didn't.

Codex

12/12, but LFM made 12 tool-argument errors, mostly integers sent as strings:

failed to parse function arguments: invalid type: string "96787", expected i32
Enter fullscreen mode Exit fullscreen mode

It retries and recovers. The cost is time: 85 s median for the bug fix, 599 s on the first run.

Claude Code

  • LFM passed the bug fix once in three runs. The other two died with: 
  Prompt is too long · automatic compaction failed: summarization produced empty response
Enter fullscreen mode Exit fullscreen mode
  • Timer page: 3/3, but an 881 s median, and one run hit the 900 s cap.
  • Qwen: 0/3. For the page and the pelican it replied that the task was complete. The working folder had no files.

An error tells you something failed. A false "done" only shows up when you check the folder yourself.

Pelicans

Called directly, LFM's pelican floats above the bike; inside an agent it actually sits on it. Qwen's drawings are more complete. Claude Code + Qwen produced no SVG.

Takeaways

  • Small local model, small context: pick the agent with the smallest prompt. Here that was pi, then Codex.
  • If you must use Claude Code locally, give it more context (Ollama's docs suggest 64K+) and look at Unsloth's notes on trimming its prompt (--bare). I didn't test those here.
  • Always ls after an agent says it's done.

Limits

  • VRAM was sampled once per second, so I only saw resident usage (~2,641 MiB LFM, ~3,111 MiB Qwen), not peaks.
  • Tool-error counts come from log keyword matching and are approximate.
  • Qwen got one run per task.
  • Three small tasks, not a real project.

Full data and configs: https://kunbuilds.com/en/posts/agents-local-model-6gb/

Sources

Top comments (8)

Collapse
 
mihai_leanzero profile image
Mihai Perdum •

The empty folder with a confident "done" is the most useful finding here. For the Qwen 0/3 runs, did the llama-server log show a tool call arriving at all, or only a plain text reply? If the model emits its call in a format the server's parser doesn't pick up, it can come back as normal content with finish_reason stop, and the agent reads that as finished. From the outside that looks the same as the model claiming work it never did, but the fix is different. Curious which one it was.

Collapse
 
kunfieldnotes profile image
KUN Field Notes •

Good question — for Claude Code + Qwen the three failures were not the same failure mode.

Task A (bug fix): agent-side error only — Prompt is too long · automatic compaction failed: summarization produced empty response. No fake "done".

Task B (timer page) and pelican: the agent printed a confident completion in plain text (timer.html / pelican.svg "done"), then stopped. Working dirs had no output files (only the starter/check files or an empty repo). Our run matrix logged tool_errors=0 for those cells (keyword match on the agent/server logs; approximate).

I did not capture the raw OpenAI-compatible finish_reason field from llama-server’s HTTP responses, so I can’t quote finish_reason=stop verbatim. What we can see: long natural-language completions after the ~14k-token first prompt, zero tool-error keywords, and an empty folder on ls. That looks like prose claiming work that never landed as a usable Write — which matches the unparsed-tool-call failure mode from the outside, but our evidence is the empty folder + plain-text "done", not a confirmed parser miss.

Fix still starts the same way: always ls after an agent says it’s done. Next time I’ll dump the chat-completion JSON (including finish_reason / tool_calls) so we can tell those two cases apart cleanly.

Collapse
 
mihai_leanzero profile image
Mihai Perdum •

That's about as far as it goes without the raw response, and the empty folder plus a plain-text "done" is a fair basis. One thing to check when you dump the JSON: look in the message content for literal tool-call markup (Qwen's <tool_call> tags or similar). If it's sitting there as text, that confirms the parser miss.

Thread Thread
 
kunfieldnotes profile image
KUN Field Notes •

Good tip, thanks. Next run I'll dump the raw completion JSON and grep the content for literal <tool_call> tags. If they're there, it's a parser miss, not the model claiming work it didn't do. I'll report which one it was.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Same local model, same tasks, 12/12 with pi and Codex, 7/12 with Claude Code. The system prompt decided that gap, so the harness belongs in the result, not in a footnote. Report it next to the model every time, or the pass counts do not compare across tools.

Collapse
 
kunfieldnotes profile image
KUN Field Notes •

Agree — and that’s the main point of the table. Same weights (LFM Q4_0 / Qwen UD-Q4_K_XL), same tasks, same llama.cpp server; only the agent (harness + system prompt + tools) changed. Pass rates were 12/12, 12/12, 7/12, and the first-prompt sizes were ~1.4k / ~6.4k / ~15.8k tokens. Without naming the harness next to the model, those counts aren’t comparable across tools. I’ll keep agent + prompt size in the result block on follow-ups, not as a footnote.

Collapse
 
benchsmoke profile image
BenchSmoke •

The empty-folder "done" case is the one I'd put at the top of any agent eval. We hit the same shape: an agent called a task finished without running its own code, and only a separate check found the bugs. Did you keep the llama-server first-prompt timings for every run, or only the medians in the table?

Collapse
 
kunfieldnotes profile image
KUN Field Notes •

Same lesson here: never trust "done" without an ls or a test run. On timings: I kept the full llama-server logs for the run matrix, so per-request prompt-eval times are all there. The three numbers in the table (5,666 / 18,567 / 48,837 ms) are the first request of one run per agent, not medians. I'll pull per-run numbers into the follow-up so the spread is visible.