I've been benchmarking local LLMs as coding agents inside my own IDE (Teaspoon IDE) on a CPU-only machine — Ryzen 5 4500U, 32 GB RAM, no GPU, Ollama.
One symptom kept showing up across models: the agent would "re-read the same file over and over," or claim "I read it, but the contents aren't visible." Classic weak-model behavior, right?
It was a bug in my own harness. This is the record of how a 4B model exposed it — and how a control experiment proved it.
The failure that started it
The task (T3): "Running pytest shows one failure. Find the cause, fix it, and confirm all pass." I gave it to qwen3.5:4b — a small, fast 4B model — on my old harness.
What happened:
-
READ_FILE: test_calc.py— reads it -
READ_FILE: calc.py— reads it -
READ_FILE: test_calc.py— reads it again. My dedup logic answers: "unchanged — use the content you already read" - The model replies: "I read them, but the contents aren't visible here"
- Tries
GREP, gets 0 hits → re-reads again → burns all 6 turns and fails
I was telling it "unchanged," and it was insisting it couldn't see the contents. My first assumption: the 4B just isn't smart enough to follow.
The investigation: tool results survived exactly one turn
Tracing the code revealed the flaw. The agent loop accumulated history like this:
- User messages → kept ✓
- Assistant replies → kept ✓
- Tool results (READ_FILE contents, command output) → present in context for exactly ONE turn after execution, then silently dropped
So:
- Step 1 reads the file → step 2 can see the content → everything works
- From step 3 on, that content has already vanished from context
- My duplicate-read suppression — "unchanged, use what you have" — was pointing at data the model did not have
The model's "I can't see the contents" was an accurate status report. It knew it had read the file (its own past READ_FILE commands were in history) but the payload was gone. It was debugging my context assembly correctly.
Bonus discovery: past evaluations needed rewriting
This changed how earlier runs looked. My 9B model (qwen3.5:9b) had been rated as "redundantly re-reading files" — 2 of 7 steps wasted on the old harness.
That wasn't weak memory. It was legitimate self-repair: fetching content that had disappeared. The smarter models were working around my bug. The weak one just reported it honestly.
The fix (and a second bug it surfaced)
- Persist tool results into running history — each step's tool output now stays in context for the whole run, so "unchanged" finally tells the truth
-
Strip outer quotes from GREP patterns — models kept emitting
GREP "member.*discount"and the literal"characters ended up in the regex, matching nothing. Seven turns of re-quoting, zero hits. The parser now strips surrounding quotes, and a zero-match GREP reply carries a hint to simplify the pattern or just READ the file
The control experiment
Same model, same task, same prompts — only the harness fix differs:
| Condition | Result |
|---|---|
| 4b · old harness | All 6 turns burned · re-read loop · failed |
| 4b · fixed | Completed in 5 turns (pytest → read → fix → re-verify, correct) |
| 9b · old harness | 7 steps with 2 re-reads (self-repairing) |
| 9b · fixed | 3 steps, zero re-reads, shortest path |
| e4b · fixed | 4 turns, zero re-reads, clean |
The same 4B model went from "hopeless" to "clean shortest-path completion" — just by fixing the code around it.
Lessons
- "Model weakness" and "harness weakness" are inseparable until you check. Every "this model is useless" article may actually be rating the harness's constraints. If you build agent tools, suspect what's in the context before you suspect the model.
- Dedup/context-saving must be designed against what the model can actually see. "It read it before" is only true while that content is physically in the prompt.
- Weak models are the best harness-testers. The strong ones paper over defects with self-repair; the 4B, with no slack, faithfully visualized my flaw. If you're testing your own agent loop, run it on a marginal model.
Data
All raw logs, task definitions, and the evaluation record are public:
- Full measured data + analysis: https://labo.mond.jp/articles/localllm-harness-bug.en.html
- Fixtures, prompts, run artifacts: https://github.com/cuculhart/ai-editor-lab/tree/main/experiments/localllm
- The harness (Teaspoon IDE, FSL-1.1-MIT, fix shipped in v1.5.1): https://github.com/cuculhart/teaspoon-ide
The environment is deliberately modest — an old 6-core laptop CPU, no GPU. If the harness had been worse, every model would have looked broken. Check your context before you check your model.
Top comments (2)
Could you separate the two fixes in a 2x2 run: old/new tool-result retention crossed with old/new GREP parsing, using the same task and inference settings? That would help distinguish recovery from missing context from recovery from a search mismatch.
A model-free regression could also check
historyForRequest: read a file containing a unique sentinel, advance two turns, then inspect the exact next request. Assert that a dedup reply saying "use what you already read" is backed by the retained file payload. Repeat after context trimming, where "previously read" can become false again.These are proposed checks, not tests run against Teaspoon. AI-assisted diagnostic suggestion.
Both good suggestions — and the second one is implementable without a model at all, which makes it the more valuable of the two. On the 2x2: fair point — the published table bundles both fixes. The trace hints at what it would show. The re-read loop is what burned the turns; the quoted-GREP miss only appeared at step 5, after the loop had already started. My prediction: (old retention, new GREP) still fails — the model can search perfectly and still not hold the file contents — while (new retention, old GREP) probably passes this task, since GREP wasn't on the critical path. The interesting variant would be a task where search is mandatory — a larger repo where the symbol has to be located, not read directly. There I'd expect the GREP cell to dominate instead. The fixtures are already public, so the cost is just CPU time; I'll add the split if I re-run. On the model-free check: agreed, and it's directly implementable. The e2e harness already mocks the model endpoint and captures the exact outgoing request bodies — the sentinel pattern is in use today, just inverted (asserting a secret never reaches the request). Your version — script the mock to emit READ_FILE twice, then assert the request after the "unchanged, use what you have" reply still contains the payload — pins a clean invariant. One detail on the trimming caveat: it can't trigger mid-run today. readCache is scoped to a single agent run, and the run's history accumulates during the loop without trimming — compaction only applies to prior turns at construction. So "previously read" can only go false across turns, where a fresh cache means no dedup reply can be issued anyway. But that's a structural accident, not a guarantee — if mid-loop trimming ever lands (longer MAX_STEPS, context pressure), the dedup claim would silently start lying. That's exactly what the test should protect. I'll write it.