Originally published on hexisteme notes.
I was running a local capture proxy in front of the vendor's API — a small man-in-the-middle process I'd set up for an unrelated token audit — when my coding-agent CLI spawned two sub-agents back to back. Same task, same working directory, same assembly path. One sub-agent got routed to a mid-tier model, the other to a small model. Because the proxy was logging full request bodies, I could diff them byte for byte. That diff is the whole essay.
The accidental A/B
The two request bodies were 615,341 bytes and 617,134 bytes. That's a difference of 1,793 bytes, about 0.3% — for practical purposes, the same payload. Same system prompt scaffolding, same tool definitions, same conversation history, same task description. Nothing about the content should have made these two requests bill differently. If token count were a property of the bytes on the wire, these two numbers should have landed within a rounding error of each other.
They didn't.
The bill
The mid-tier-model request was billed 246,525 input tokens. The small-model request, for a body 0.3% larger, was billed 196,892 input tokens — 49,633 fewer, a 20.1% reduction. Expressed as tokens per byte, that's 0.390 for the mid-tier model against 0.310 for the small model. Same text, essentially the same byte count, and one model's meter reads a fifth lower than the other's for it.
The two Stop hooks behind this note are on GitHub under MIT: hexisteme/hard-gate-hooks. They ship with their tests and a read-only scanner that prints what they did on **your* machine, not mine — including the case where it tells you they aren't worth wiring up yet. No email, no signup.*
I want to be precise about what "billed" means here, because it's easy to round this into something looser. The input-token figure I'm comparing is the accounting field the API actually returns in the response — the number that determines what the request costs. It isn't an estimate I computed from the bytes; it's the number the vendor's own usage accounting assigned to each request, for near-identical input.
What this means — and what it doesn't
The clean version of the claim is: a token count is not a property of a request. It's a property of the pair (request, model). The same bytes, tokenized or accounted for under a different model's scope, produce a different number.
Here's where I want to slow down and be honest about the limits of what I actually measured. What I have direct evidence for is the accounting difference itself — 20.1% fewer tokens for the same bytes, measured once, on one pair of requests. The most plausible mechanism behind that gap is that the two models use different tokenizer vocabularies, so the same run of text segments into a different number of pieces. That's the standard explanation for this class of effect and it fits what I saw. But I did not run an experiment that isolates the mechanism. I didn't tokenize the same string offline with each model's tokenizer and count pieces directly; I didn't rule out that some of the gap comes from a difference in what gets counted into the input-token field in the first place — cache-eligible content, tool-definition overhead, or some other accounting-scope difference on the vendor's side rather than the tokenizer itself. So the correct claim is narrower than "different tokenizer, confirmed": it's "the accounting differs by model, and a tokenizer difference is the leading candidate explanation, unconfirmed."
That distinction matters more than it sounds like it should, because the practical implications below hold either way. Whether the mechanism is tokenizer vocabulary or accounting scope, the meter itself is model-scoped. That's the load-bearing fact, and it's the one I actually measured.
Implications
Cross-model "$ per token" comparisons don't compose. If you're comparing the cost of running the same workload on two models by multiplying a price-per-million-tokens figure by a token count, you're implicitly assuming the token count is the same across models for the same work. It isn't. A 20% gap in the denominator, on top of whatever gap exists in the price-per-token numerator, means the two models' effective prices for identical work can diverge from what a naive "$/Mtok × tokens" comparison suggests. If you want a comparison that actually composes across models, normalize by something model-independent — bytes of input, or a fixed task definition — not by the token count either model reports.
"Token savings" measurements are only valid within one model. If you're running an optimization — trimming a system prompt, restructuring tool definitions, cutting a stale cache block — and you're measuring the win in tokens saved, that number is only meaningful as long as the model stays fixed across the before/after comparison. Switch models in the middle of a measurement campaign and the accounting scope shifts under you; part of whatever delta you see is now optimization, and part is just a different meter. I've made this mistake before in a different form — comparing pooled metrics across roles that weren't actually comparable — and this is the same shape of error one layer down: the unit of measurement quietly changed between the two things being compared.
A cheaper model's effective unit price can be better than its price sheet implies. If a small model both charges less per token on the price sheet and meters fewer tokens for the same bytes, its actual cost advantage for a given task is larger than the sticker prices alone would suggest. Conversely, if a model's tokens-per-byte ratio for your workload runs high, part of what looks like the price sheet's "premium" is actually the meter charging more units for the same input, not just charging more per unit. Either way, the sticker price alone doesn't tell you the effective cost of a task — you need the accounting ratio too, and that ratio is workload-dependent as much as it's model-dependent.
Honest limits
This is n=1 pair. One measurement, one moment, one workload. I'd treat the 20.1% figure as "this magnitude of gap exists and is worth checking for," not as a constant you can plug into a cost model for any pair of models.
The measurement ran behind a capture proxy, and that matters for the absolute numbers: request assembly behind this kind of proxy setup runs roughly 3x larger than it would in normal operation, because a caching optimization that's normally active gets disabled when a proxy sits in the path. So don't read "615,341 bytes" as a typical request size for a sub-agent spawn — it isn't. What keeps the comparison valid despite that inflation is that both requests took the identical assembly path, behind the identical proxy, with near-identical bodies. The absolute byte counts are inflated; the ratio between the two requests is not, because whatever inflated one inflated the other by the same mechanism.
The payload itself was dominated by English-language system and tool text — scaffolding, tool schemas, instructions — not code and not non-ASCII text. Tokenizer vocabulary gaps between models are known to vary by content type: code, non-English text, and structured data can all tokenize differently than prose does. So the 20.1% figure and the 0.390-vs-0.310 tokens-per-byte ratio are specific to this kind of payload. A code-heavy request or a request dominated by a non-Latin-script language could show a very different ratio, in either direction. I haven't measured either of those cases, and I'm not going to guess at what they'd show.
What I'd want before trusting this number as a general rule: the same near-identical-bytes comparison repeated across a few different payload types — code-heavy, non-ASCII-heavy, and prose — across a few different model pairs, without a proxy in the path so the absolute sizes are representative too. Until then, the finding stands as exactly what it is: one clean natural experiment showing that the meter is model-scoped, with a plausible but unconfirmed mechanism, and a strong reminder to stop assuming the token denominator is a fixed property of the request when comparing costs across models.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (4)
The (request, model) framing is the right mental model, and it has a second-order consequence worth flagging: it breaks context-window budgeting too, not just billing. If you size a prompt against the tokenizer of the model you usually route to, then fail over to a different model under load, the same bytes can quietly cross that model's context limit even though your pre-flight count said you were fine. We hit this exact edge routing sub-agents between a mid-tier and a small model — the small one's cheaper meter is great until its window is also smaller and a payload that "fit" gets truncated silently.
One question on your measurement: was the 20% roughly stable across payload composition, or did it move with the mix of code vs prose vs whitespace? Different vocabularies tend to diverge most on code and non-Latin text, so a single diff on one assembly path might understate the variance. Would love to see the spread across a few hundred requests.
It isn't a spread — it's n=1, and you've found the right seam. The post says so outright ("This is n=1 pair. One measurement, one moment, one workload"), and it names your exact suspicion as its limit: the payload was dominated by English system and tool scaffolding, not code and not non-ASCII, so the 0.390-vs-0.310 tokens-per-byte ratio is specific to that payload type, and a code-heavy or non-Latin request could move it in either direction.
What I can add is what happened when I went to look after your comment. The spread is not computable retrospectively. The capture proxy wrote request and response bodies to a session-scoped scratchpad; that session is gone and so is the directory, so no payload text survives to classify by composition. The obvious fallback doesn't rescue it either — my transcripts carry over 330,000 turns with
input_tokensand a model field, which is plenty of scale, but there is no request-byte field and the persisted content per turn is only the incremental delta, never the assembled wire payload. Tokens at scale, bytes nowhere. Getting your few hundred requests means re-running the capture across code-heavy, prose, whitespace-heavy and non-Latin payloads, which is the follow-up the post names as unrun. Until then, one point is the honest way to read the 20.1%.The failover half is the sharper part of your comment and it isn't in the post. One thing it adds: the direction of the token ratio and the direction of the window size are independent. In my pair the small model billed 20% fewer tokens for the same bytes, so a failover downward looks safe on the meter — while its window is the smaller one. Cheaper per byte and less room, from the same swap. You cannot infer fit from the cheaper meter, which is exactly the silent truncation you hit.
Real data point that confirms the "normalize by bytes" implication - because the wire has byte limits that tokens don't.
In the agent harness I run, we shipped a feature that persisted a model's reasoning trace and passed it back into the next request's context (thinking-mode pass-back). It worked until the day the request body hit 2.5MB of reasoning text and the vendor returned HTTP 413 request-too-large against a ~3.4MB body limit. The interesting part: nobody had budgeted for that, because everyone was budgeting in tokens. Token-wise the payload was well within limits; wire-wise it wasn't - and the two meters scale differently per model, exactly as you describe. A model with a lower tokens-per-byte ratio for the same workload gives you more bytes per token, so the same token budget produces a larger request body and sits closer to the wire limit.
The fix wasn't token-budgeting, it was reverting the pass-back. What stuck with me is the asymmetry: billing meters are token-scoped, request limits are byte-scoped, and only one of them follows the (request, model) pair. So for cross-model routing, byte-normalization isn't just a cost-comparison nicety - it's the only budget that holds on the wire at all.
The asymmetry is the part worth keeping: billing follows the (request, model) pair, the wire limit follows the request alone. Your 413 is the cleanest illustration of what that costs, because the failure is invisible to the meter everyone is actually watching.
My own measurement environment demonstrates the same hazard from the other side. The two requests I compared were 615,341 and 617,134 bytes — but those absolute figures were roughly 3x inflated over normal operation, because putting a capture proxy in the path disabled the CLI's tool-schema deferral and pushed the full catalog into every request. Nothing about the task changed. A transport-layer decision multiplied the request body, and it would have been invisible to any token budget I was keeping. Same class as a reasoning trace passed back: bytes moved for a reason nobody budgeted, and only the wire noticed.
Where I'd push slightly. Byte-normalization is the only budget that holds on the wire, but it does not hold on the context window, which is token-scoped and model-scoped at once. Another commenter on this post hit the mirror failure — failing over to a cheaper model whose tokens-per-byte is lower but whose window is also smaller, and getting silent truncation on a payload that "fit." So cross-model routing needs two meters that don't substitute for each other: bytes against the wire limit, and tokens under the target model's own tokenizer against that model's window. Your revert is the right fix for the first. It buys nothing against the second.