I tried to replace the KV cache with a function. The model started inventing numbers.

Oleksandr Tepliuk · October 2026

Three experiments on a frozen open model, run on a desktop CPU. Each protocol and decision rule was
written before the run; code and raw results are listed at the end.

Every new token a language model generates looks back at all previous tokens through the KV cache. For
a text the model has already read, that look-back is a fixed function of the new query. So I asked: can
we fit a small function to it once and throw the cache away?

• E0, function alone: no. On Qwen3-1.7B, with 20 facts hidden in a ~2,000-token text, full attention recalls 82.5% of the facts. A fitted function using 1/8 of the cache's memory recalls 0.8%. Even at 1/4 of the memory, nothing tested got above 14%.
• The model does not say "I don't know". It confidently names a plausible wrong number: "Locker 8957" instead of 9550, or "2345", "1001", "9876".
• E1, function plus 8 exact records per query: the function has a role, but not the one I expected. It does not remember facts. It corrects bad selection. With crude sign-bit selection of the 8 records, it lifts recall from 7.5% to 30.8%.
• E2, cheap but accurate selection. Keys compressed to 16 dimensions, then a shortlist of 64 is re-checked exactly and the top 8 are kept. This alone recalls 82.5%, the same as full attention, while reading about 8% of the cache per token. That recipe is already known (it is close to ShadowKV), so the win is not mine. The function on top adds only +5 pp, which is not significant, but it halves the error on verbatim copying.

The question

The KV cache stores a key K and a value V for every token read. At each step the new query q is
multiplied by all stored keys, and the output is a softmax-weighted sum of the values. For a text that
is already read, this is a function o(q). The idea was to fit a compact f(q) ≈ o(q) once, after reading,
and drop K and V. Memory would stay constant, and the cost per token would no longer grow with the
length of the text.

This is close to, but different from, known work:

• Models trained to fold the past into a fixed state: linear attention, Mamba, Gated DeltaNet, TTT/Titans.
• Conversion of trained models with extra training: LoLCATs, Cartridges.
• Training-free cache compression: FourierAttention, Expected Attention, KQ-SVD.

Here the model stays frozen. The function is fitted per text, in one closed-form step (ridge regression),
to reproduce the attention output on simulated future queries. Those queries are the text's own queries,
rotated by RoPE to future positions, so nothing from the future leaks in.

Setup

• Model: Qwen3-1.7B, fp32, on a 6-core desktop CPU. I chose it because it is a recent small model with full attention in all 28 layers. Newer hybrid models already replace most layers with linear state, which would blur the test.
• Texts: 6 texts of ~2,000 tokens of reference prose. Each text has 20 invented facts of the form "name → 4-digit number", in 5 templates, placed at random.
• Metrics:
  • recall — the share of 120 questions ("Question: … Answer:") where the right number appears in 8 greedy tokens;
  • copy NLL — 128 tokens repeated from the middle of the text;
  • new-text NLL — prose the text did not contain.
• What gets replaced: the read text, except 4 "sink" tokens. The question tokens are always attended exactly. Approximate and exact parts are merged exactly in log-space.
• Code check before every run: an arm that goes through the same replacement path but computes the text exactly must match full attention. It did, with a maximum logit difference of 4–8·10⁻⁵.

E0: a function instead of the text

Arms at an equal memory budget, per KV head:

• recent: the last tokens only;
• centroid: k-means of the keys, with summed counts and mean values;
• ffit: the same centroids, but the values are fitted by ridge regression to the attention output;
• frand: random anchors with fitted values; stores no keys at all.

The fitted function is the best compressed arm on copying, so fitting beats plain averaging. But exact
recall is gone. A fact lives in a few specific tokens, and any smoothing spreads them out. New-text NLL
barely moved (2.70–2.78). That test is weak, because the new prose does not depend on the text.

The behaviour that worried me most is the failure mode: compressed memory turns into confident
fabrication. The model still knows a 4-digit number is expected, so it produces one.

E1: function plus a few exact records

Next, the full cache stays in RAM. For every query, in every layer and head, k exact records are added
next to the function. The rule for the mix was fixed in advance: exact records carry their own mass, and
the function covers only the remainder, so no record is counted twice. The key control is the same k
records without the function.

• With sign-bit selection the function adds +23 to +28 pp (k = 8 and 16), p < 0.001. Copying also improves in all 6 texts.
• With exact selection it adds +11.7 pp at k = 8 (p = 0.024, which does not pass the 4-pair correction). At k = 16 it adds nothing: 16 exact records already match full attention.

Interpretation. When the selected records are wrong, attention without the function piles all its
weight onto them. The function restores the background mass of the rest of the text, so wrong picks do
not dominate.

Side finding: sign bits are a poor selector for attention. When the 8 or 16 records are chosen by
the sign of each key dimension alone, recall is 7.5–12.5%, against 73–83% when they are chosen exactly.

E2: cheap but accurate selection

Selectors, each with and without the function, all ending with 8 exact records (Quest: 4 pages of 16):

"Reads per token" is computed per head against the full K+V of the text. It is arithmetic, not a timing
measurement.

What I take from it:

• The re-check is the part that matters. Low-rank selection alone reaches 58%; the exact re-check of 64 candidates lifts it to 82.5%.
• You do not need to find every important record. On the first text, low-rank + re-check finds only 68% of the true top-8 records, yet recall matches full attention. The missed records probably duplicate each other: several tokens of one fact, or several heads looking at the same place.
• The function rescues weak selectors (+22 to +26 pp for sign bits and Quest, p < 0.001). With a good selector it adds only +5 pp of recall (p = 0.31), but it nearly halves copy NLL (0.24 → 0.135). It restores soft, verbatim fidelity that 8 records cannot.
• Quest with 4 × 16 pages is too coarse for facts that span a few tokens. The real Quest takes more pages; I did not tune it.

What is new here, and what is not

• Not new: the winning recipe. Low-rank keys plus a re-check of a shortlist is close to ShadowKV; Quest, MagicPIG and RetrievalAttention are relatives. I only confirmed it on another model.
• Not new: sparse attention in general. New models increasingly ship with a trained selector (DeepSeek's DSA and others), and llama.cpp is adding kernels for them.
• What I have not seen stated this way:
  1. A per-text fitted function cannot replace the cache on a frozen model. It holds the gist, not the facts, and the model fills the gap by fabricating with confidence.
  2. The same function works as a corrector for selection errors and as a booster of verbatim fidelity.
  3. Sign-bit keys select attention targets very poorly. In my earlier document-retrieval tests, sign-bit embeddings only fell behind beyond ~20,000 records.
• What I could not find implemented: post-hoc selection like this for ordinary dense models on CPU in mainstream local runtimes. I checked only partly.

Takeaways if you run models locally

• Do not smooth the KV cache. Approximations that look fine on perplexity can wipe out exact recall.
• When you compress memory, measure "I don't know" separately. The failure is not silence; it is a confident wrong answer.
• For selective attention, budget for a re-check step. It is the cheapest part and gives most of the quality.
• On CPU, all of this matters only for long contexts. At 2,000 tokens the KV cache of Qwen3-1.7B is about 7% of what the CPU reads per token (the weights dominate), so even a 90% KV saving gives only a few percent of speed.

Limits

• One small model, texts of ~2,000 tokens, synthetic facts.
• In E1 and E2 the function was tested at two sizes only (1/8 and 1/16 of the cache).
• Reads are computed, not timed. Selection is simulated using the full keys.
• Text 6 was hard for every arm, including full attention (11 of 20).

Reproducibility

In the lab repository, experiments/kv_function_v0:

• PROTOCOL_UA.md, run.py, results.jsonl, RESULT_UA.md — E0;
• E1_PROTOCOL_UA.md, e1.py, e1_results.jsonl, E1_RESULT_UA.md — E1;
• E2_PROTOCOL_UA.md, e2.py, e2_results.jsonl, E2_RESULT_UA.md — E2.

Each protocol states the decision rule and was written before its run; the protocols and reports are in
Ukrainian.