RTX 4070 + logprobs: 96 of 100 Claude Code Requests Needed the Cloud. 97 of 200 Steps Could Go Local

Routing whole requests to a local LLM did not work for my Claude Code workload. I took 100 jobs from my work log (March to September 2026), rewrote each one as the instruction I would have typed, and labelled where it could run. 96 needed a frontier model. The other 4 had to stay local because they contained personal or financial details, not because a local model could do them. Zero were simply "a local model is good enough."

Then the unit went down to single steps inside agent sessions. Out of 200 tool-call steps pulled from my Claude Code logs, 97 were labelled "a 4B to 9B local model is enough if you hand it the material." That is 48.5%.

The step-level router that came out of this is on GitHub under MIT: kenimo49/local-step-router. This post covers the numbers behind it, what the router code actually looks like, and where a logprobs judge on an RTX 4070 helped and where it did not.

Why request-level routing failed

A request like "fix the flaky test" is a label, not a payload. Nothing in the text tells a local model what to do until something reads the repo. In the 100 reconstructed requests, the task mix was mostly agent work (74 of 100), and agent work is exactly what you hand Claude Code in the first place.

Raw instructions told the same story. Of 150 instructions sampled from interactive sessions, 138 only made sense with the earlier conversation ("send it", "do the daily log"), and only 5 were labelled local-capable. Of 100 fixed prompts that my cron pipeline sends to claude -p, also 5.

Which steps go local

The 200 steps came from three tool types in the logs. The split is lopsided:

Almost all of the local share is WebFetch. A fetch prompt usually says what to pull out ("the price", "the release date"), and once the page text is in hand, that is extraction. Commit messages go local only when they are one-line summaries of a diff. Sub-agent delegation is research and multi-step tool use, and it stays frontier.

Two caveats matter here. The labels are judgments, not runs: "local is enough" means "given the page body or the diff." And they were assigned by my AI assistant, with my review planned only for disagreements.

So I ran a sample. 20 of the 86 local-labelled WebFetch steps went to qwen3.5:9b on Ollama (think=false, temperature 0), with the page re-fetched and extracted by trafilatura. 11 could be compared with the original output: 8 matched, 2 were partly wrong (a merged-cell spec table read into the wrong column, a title reported as missing), and 1 was wrong (it quoted "about 1 to 3 months" as evidence and still answered yes to "does it say 2 to 3 months?"). The other 9 never reached a fair comparison: JS-rendered pages with no extractable text, pages that had disappeared, or an original fetch that had already failed. Local processing took 6 to 66 seconds per item, median about 18. Most of the loss happened before the model, in fetching and extracting the page.

The router: rules first

The repo's Router checks rules top to bottom, first match wins. A probability judge only runs for steps no rule matched, and it is off by default. This is the shipped examples/routing.yaml, rules part:

rules:                      # step rules, top to bottom, first match wins
  - match: {tool: WebFetch}
    route: local
  - match: {tool: [Agent, Task]}
    route: frontier
  - match: {kind: commit_message}
    route: frontier         # frontier until a probability judge is added

probability: null           # off until measured with the same judge model (chapter 14)
default_route: frontier

And from Python:

from steprouter import config
from steprouter.router import Router

router = Router.from_config(config.load("examples/routing.yaml"))

router.route({"tool": "WebFetch", "prompt": "Extract the price table"})
# Decision(route='local', by='rule', p_local=None)
router.route({"tool": "Edit"})
# Decision(route='frontier', by='default', p_local=None)

Decision.by tells you whether a rule, the probability judge, or the default made the call.

Why rules first: on the 444 non-sensitive items across all three sets (107 labelled local-capable), I compared a single rule against asking a local model for P(local is enough):

The 9b judge's AUROC of 0.809 was mostly the judge recognising the step type. Within one type it could barely separate items: 0.095 on the pipeline prompts, 0.411 on raw instructions. If the step type decides, write a rule. Keep the probability for what the rules leave open.

I also tried qwen3.5:4b as the routing judge. It timed out under GPU contention at 414 of 450 items, so it has no routing result.

Reading logprobs from Ollama

Where the local model does earn its place is the entrance gate: "does this request contain actual sensitive data?" The judge asks for a one-letter answer (Y or N), generates exactly one token, and reads the candidate letters from top_logprobs. In the repo this is steprouter/judge.py, using the native /api/chat with think: false, num_predict: 1 and top_logprobs: 20. It returns the probabilities renormalised over the candidates, plus mass: how much probability landed on the candidates at all.

mass is the field to watch, because two things make it collapse without any error:

• The first token is not the answer. A model that reasons before answering spends token 0 on its reasoning. With think: false, qwen3.5 answers in the first token. gpt-oss:20b did not put a label there and gave mass 0 on my setup, so I could not use it this way. The other models I tried stayed between 0.89 (qwen3.5:0.8b) and 1.00.
• A candidate outside the top 20 reads as 0. Ollama rejects top_logprobs above 20 (the check in server/routes.go; a PR to raise it to 100 was closed without merging). If a candidate is not in that list, its probability is simply missing, and the others share 100%.

The repo's judge marks a result unusable when mass is below 0.5. A first test on any new judge model: send one known question and check that mass is close to 1.

One correction to something you may have read, including in an earlier version of my own notes: the OpenAI-compatible /v1/chat/completions endpoint does return logprobs. The v0.12.11 release notes say both APIs support them, and in issue #16117 a maintainer showed it working before the reporter closed it as their own client bug. The compatibility page in the docs still marks logprobs unsupported. The repo uses the native endpoint, so it does not depend on which of those is current.

The threshold is not 0.5

qwen3.5:4b ranked the 60 English test requests almost perfectly for sensitive content (AUROC 0.999), but its P(Y) stayed low. With a fixed 0.5 line it missed all 20 sensitive requests. Drawing the line from labelled examples with 5-fold cross-validation brought that to 2 misses and 1 false alarm. On the Japanese version of the same set, the fixed line missed 9 of 20 and the calibrated one missed 1.

The repo's CLI does this from your own judged examples:

python3 scripts/judge_set.py qwen3.5:4b en > judged.jsonl
steprouter-threshold judged.jsonl --margin 0.3

It prints JSON with auroc, the line (lowest P among your positive examples times the margin), and 5-fold cv counts of misses and false alarms, and it warns when AUROC is under 0.95. The data set is 60 made-up requests, so treat these as small-sample numbers, and redraw the line on your own labels.

Where the RTX 4070 fits

All of the judging above ran through Ollama on one RTX 4070 12 GB (version 0.20.2 for the 60-request tests). Small judges were not faster on that box: qwen3.5:0.8b through 9b all took about 0.5 to 0.7 seconds for two questions, mostly network round trip, while qwen3.5:35b-a3b took about 3.3 seconds because it does not fit in 12 GB. If you want a 35B MoE model on the same card for heavier local work, the --cpu-moe flag in llama.cpp took it from 12.2 to 34.6 tok/s in a separate benchmark. That benchmark is about speed; the routing numbers above did not use it.

What I would copy

1. Count your own steps before tuning anything. My 48.5% is almost entirely WebFetch. A workload with fewer fetches will get a smaller share.
2. Decide by step type first. One WebFetch rule beat both probability judges in my data.
3. Check mass, then draw the threshold from your own labels. A fixed 0.5 missed every sensitive English request with qwen3.5:4b.

The code: kenimo49/local-step-router. MIT, Python 3.10+, PyYAML as the only dependency, Ollama 0.12.11 or later for the judge. The tests run without Ollama.

The full write-up, with the step labelling criteria, the sensitive-data gate, and the typed return values, is the book this repo accompanies: Local LLM or Claude Code? Kindle, included in Kindle Unlimited.