At 02:14 the serving page stays oddly quiet.
Your node still reports twelve percent CPU usage.
Queue depth reads zero on the local listener.
Token counters climb without a matching wait queue.
Prompt tokens outrun completions by a wide margin.
Nightly eval deadline slack shrinks on every call.
Which operational action follows from that evidence?
You do not add another free replica yet.
You stop the cache-busting prompt prefix first.
The metric contradiction
Idle CPU is not proof of spare capacity.
It often hides a prefill tax on every call.
Each request looks unique to the server cache.
A timestamp in the system prompt is enough.
A request id in the tools preamble is enough.
The prefix hash then changes on every call.
You pay input tokens for the same instructions.
You wait for prefill on text you already sent.
Queue age stays low because each job is short.
Free capacity is the wrong bet in this shape.
The node is not saturated on compute cycles.
The token budget is what actually breaks first.
Scaling the free node will not fix the prefix.
Topology you can reproduce locally
Run the whole drill on one workstation only.
Keep production traffic out of this path.
-
client: serial eval runner under your control -
admitd: admission proxy bound to127.0.0.1:8088 -
server: local free model server behind the proxy -
metrics: JSONL file with one event per request
You can stage the server side with MonkeyCode.
Free model access and a free server option exist for lab work.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Treat that path as a rehearsal environment only.
Do not read it as an availability SLO.
Do not read it as a production capacity plan.
Declared lab conditions
These figures are labels for the drill only.
They are not product benchmarks or hardware claims.
- Workload: 40 serial jobs with an eight second deadline
- Prefix: about 1,200 tokens of stable instructions
- Question: about 80 tokens of unique user text
- Noise: an ISO-8601 clock field inside the prefix
- Cache: SHA-256 prefix digest in SQLite, fifteen minute TTL
- Pass rule: reject noisy prefixes before tokens move
Why the free node looks healthy
Prefill work is bursty and then gone quickly.
CPU averages hide the token spike completely.
Queue depth never accumulates for these tiny jobs.
You will see three fields disagree at once.
-
queue_age_msstays near zero during the window -
cpu_pctstays in the low teens on average -
prompt_tokens_totalrises in lockstep with wall time
That pattern is not a scale-out signal.
It is an admission and reject signal.
You should shed the noisy prefix right now.
You should not add more free capacity yet.
Decision table for the page
Copy this table into the runbook.
Use it before you touch replica counts.
| Observed fields | Inference, labeled as inference | Action |
|---|---|---|
queue_age_ms low, cpu_pct low, prompt tokens rising |
Cache-busting prefix, not saturation | Reject or strip the clock |
queue_age_ms high, cpu_pct high |
True saturation on the node | Shed load; skip the free path |
queue_age_ms high, cpu_pct low |
Blocked IO or a stuck worker | Debug the server; do not enqueue |
deadline_slack_ms under 1,500 |
The job cannot finish in time | Reject; do not spend tokens |
Separate observation from inference in every log line.
The first column stays observed telemetry only.
The second column is a working inference only.
Fingerprint the prefix, not the full body
You cannot meter what you cannot name yet.
Hash only the stable instruction prefix today.
Drop clocks, UUIDs, and trace ids first.
# prompt_fingerprint.py — lab helper, not a production library
import hashlib, json, re, sys
CLOCK = re.compile(
r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?"
)
IDS = re.compile(
r"\b(?:req|trace|run|job)[-_][A-Za-z0-9-]{6,}\b",
re.I,
)
def stabilize(text: str) -> str:
text = CLOCK.sub("<TS>", text)
text = IDS.sub("<ID>", text)
return " ".join(text.split())
def fingerprint(prefix: str) -> str:
body = stabilize(prefix).encode("utf-8")
return hashlib.sha256(body).hexdigest()[:16]
if __name__ == "__main__":
payload = json.load(sys.stdin)
prefix = payload["prefix"]
print(json.dumps({
"fp": fingerprint(prefix),
"stable_len": len(stabilize(prefix)),
"stripped_clock": prefix != stabilize(prefix),
}))
Feed it a prefix from standard input.
Confirm two clock-shifted calls share the same fp.
If they differ, your scrubber is still too weak.
printf '%s\n' '{"prefix":"Session 2026-09-08T02:14:03Z. Grade tests."}' \
| python3 prompt_fingerprint.py
printf '%s\n' '{"prefix":"Session 2026-09-08T02:16:11Z. Grade tests."}' \
| python3 prompt_fingerprint.py
Labeled expected output is a shared fp value.
stripped_clock should read true on both calls.
Do not proceed until those two hashes match.
Admission proxy with a hard reject
The proxy estimates tokens with a cheap heuristic.
It does not call the model for that estimate.
Four characters per token is the labeled lab rule.
Policy is reject-first on a raw wall-clock field.
You want callers to stop sending clocks.
Hiding the clock in a proxy forever trains bad clients.
# admitd.py — labeled local behavior, unexecuted in this article
import json, sqlite3, time
from http.server import BaseHTTPRequestHandler, HTTPServer
from prompt_fingerprint import fingerprint, stabilize
DB = "admit.sqlite"
DEADLINE_MS = 8000
MAX_PROMPT_TOKENS = 1500
CACHE_TTL_S = 15 * 60
def db():
conn = sqlite3.connect(DB)
conn.execute(
"CREATE TABLE IF NOT EXISTS prefix("
"fp TEXT PRIMARY KEY, hits INT, first_ts REAL)"
)
return conn
def estimate_tokens(text: str) -> int:
return max(1, len(text) // 4)
class Admit(BaseHTTPRequestHandler):
def do_POST(self):
n = int(self.headers["Content-Length"])
body = json.loads(self.rfile.read(n))
prefix = body.get("prefix", "")
queued_ms = int(body.get("queue_age_ms", 0))
slack_ms = DEADLINE_MS - queued_ms
fp = fingerprint(prefix)
ptok = estimate_tokens(prefix)
now = time.time()
noisy = prefix != stabilize(prefix)
decision = "allow"
reason = "stable_prefix"
if ptok > MAX_PROMPT_TOKENS:
decision, reason = "reject", "prompt_tokens"
elif slack_ms < 1500:
decision, reason = "reject", "deadline_slack"
elif noisy:
decision, reason = "reject", "cache_busting_clock"
conn = db()
if decision == "allow":
row = conn.execute(
"SELECT hits, first_ts FROM prefix WHERE fp=?",
(fp,),
).fetchone()
if row is None:
conn.execute(
"INSERT INTO prefix VALUES (?,?,?)",
(fp, 1, now),
)
else:
conn.execute(
"UPDATE prefix SET hits=hits+1 WHERE fp=?",
(fp,),
)
conn.commit()
event = {
"ts": now,
"fp": fp,
"queue_age_ms": queued_ms,
"deadline_slack_ms": slack_ms,
"prompt_tokens_est": ptok,
"decision": decision,
"reason": reason,
"cpu_pct_observed": body.get("cpu_pct"),
}
with open("admit.jsonl", "a") as f:
f.write(json.dumps(event) + "\n")
code = 200 if decision == "allow" else 429
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(json.dumps(event).encode())
if __name__ == "__main__":
HTTPServer(("127.0.0.1", 8088), Admit).serve_forever()
Bind the handler only to localhost addresses.
Do not expose this handler on a public interface.
It is an admission drill, not an auth layer.
Failure injection you can run
Start the proxy on the lab loopback interface.
Leave the model server stopped on purpose here.
You are testing reject behavior, not generation quality.
python3 admitd.py &
echo $! > admitd.pid
# Job A: stable prefix, should allow
python3 - <<'PY'
import json, urllib.request
body = {
"prefix": "You grade unit tests. Return JSON only.",
"question": "Does test_foo assert timeout?",
"queue_age_ms": 40,
"cpu_pct": 12
}
req = urllib.request.Request(
"http://127.0.0.1:8088/",
data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"},
)
print(urllib.request.urlopen(req).read().decode())
PY
# Job B: clock inside prefix, should 429
python3 - <<'PY'
import json, urllib.request
body = {
"prefix": "Session 2026-09-08T02:14:03Z. You grade unit tests.",
"question": "Does test_foo assert timeout?",
"queue_age_ms": 40,
"cpu_pct": 12
}
req = urllib.request.Request(
"http://127.0.0.1:8088/",
data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"},
)
try:
urllib.request.urlopen(req)
except Exception as e:
print(type(e).__name__, e)
PY
# Job C: slack already gone, should 429
python3 - <<'PY'
import json, urllib.request
body = {
"prefix": "You grade unit tests. Return JSON only.",
"question": "late job",
"queue_age_ms": 7000,
"cpu_pct": 12
}
req = urllib.request.Request(
"http://127.0.0.1:8088/",
data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"},
)
try:
urllib.request.urlopen(req)
except Exception as e:
print(type(e).__name__, e)
PY
Inject one extra retry of Job B after the 429.
Keep max_retries=1 on this lab path.
A second clocked prefix should still not reach a model.
Labeled expected output
Job A should return HTTP 200 for allow.
The reason field should read stable_prefix now.
Estimated prompt tokens should stay under 1,500.
Job B should return HTTP 429 almost immediately.
The reason field should read cache_busting_clock here.
No model bytes should move for that call.
Job C should return HTTP 429 for slack.
The reason field should read deadline_slack here.
Queue age, not CPU, drove that reject.
Read the JSONL log after the three jobs.
python3 - <<'PY'
import json
from collections import Counter
rows = [json.loads(l) for l in open("admit.jsonl")]
print(Counter((r["decision"], r["reason"]) for r in rows))
print("max_queue_age_ms", max(r["queue_age_ms"] for r in rows))
print("max_prompt_tokens_est", max(r["prompt_tokens_est"] for r in rows))
PY
Expected lab histogram for this exact script:
-
("allow", "stable_prefix")count equals one -
("reject", "cache_busting_clock")count equals one -
("reject", "deadline_slack")count equals one
If allows outnumber rejects, the scrubber failed.
If CPU was your only page, you missed the burn.
Operational threshold and rationale
Pick one reject threshold and write it down.
Do not page on utilization alone.
Recommended lab threshold for this drill:
- Reject when
deadline_slack_mssits under 1,500 - Reject when estimated prompt tokens exceed 1,500
- Reject when a wall-clock token remains in the prefix
- Ignore CPU below 30% unless queue age also rises
The rationale stays local to this lab path.
Queue age tracks wait the user will feel.
Utilization tracks busy time you already spent.
Deadline slack tracks whether the job can still finish.
Low CPU plus rising prompt tokens is waste.
High CPU plus rising queue age is saturation.
Only the saturation case argues for more capacity.
Token retries make the waste case worse.
Each retry resends the busted prefix in full.
Cap retries at one on this admission path.
Retrying a 429 from cache_busting_clock is wrong.
Fix the caller, then send one clean prefix.
Cleanup and rollback
Stop the proxy after you capture logs.
Drop the lab database before the next drill.
Restore the eval runner to its prior URL.
if [ -f admitd.pid ]; then
kill "$(cat admitd.pid)" || true
rm -f admitd.pid
fi
rm -f admit.sqlite admit.jsonl
unset ADMIT_URL
export CLIENT_URL="${SERVER_URL:-http://127.0.0.1:8090}"
echo "rolled back to $CLIENT_URL"
If a clock already shipped in production prompts, strip it there first.
Then reopen traffic through the proxy for one window.
Keep the 429 path on during that single batch window.
Rollback is a URL swap plus a process kill.
It is not a cluster drain in this drill.
Do not leave admitd running after the rehearsal.
Limitations
The four-characters-per-token rule is crude.
It will misread code and CJK text.
Use it only as an admission fence.
The clock regex will miss custom date formats.
UUIDs in retrieved documents can still bust the cache.
Fingerprint the instruction prefix, not retrieved chunks.
This drill does not measure model quality at all.
It does not prove cache hits on a vendor stack.
It only stops noisy prefixes from spending tokens.
SQLite is not an HA cache under load.
One forgotten reader can stall checkpoint writes.
That failure mode belongs to a different note.
Who should not use this approach
Do not use this proxy as an internet gateway.
It has no authentication and no TLS.
Localhost binding is a hard requirement here.
Do not use it for interactive chat with tight latency.
A reject is better than a late eval job.
A reject is worse than a late chat token.
Do not use free capacity when the job is user-facing.
Deadline slack under one prefill needs a reserved path.
Free lab nodes are for rehearsal and shed-able batches.
Do not share one listener between serving and eval.
Eval sidecars will stamp clocks into prefixes.
That mix recreates the 02:14 contradiction exactly.
Skip this drill if your prompts are already stable.
Skip it if you cannot read token counters today.
Fix telemetry first, then add an admission fence.
What you do next
Watch prompt tokens against queue age for one window.
If they diverge, reject the prefix before you scale.
Run the fingerprint drill on your lab node before the next batch.
Top comments (0)