Originally published on hexisteme notes.
"Will it fit on our hardware?" is the wrong first question. It's the one everyone asks, because
it's free to answer — the thing either loads or it doesn't.
Throughput costs you a measurement. So the capacity gate passes, and it feels like the
decision is made.
The measurement
Mac Mini M4, 24GB unified memory, ~120GB/s memory bandwidth. A 27B model, IQ4_XS quantized,
15GB on disk.
Capacity gate: pass. Metal's recommendedMaxWorkingSet is 17.76GB, the model is 15GB,
ollama ps reports 100% GPU resident. No swap, no spillover. By every "does it fit" criterion
this is a clean win.
Generation: 5.6 tokens/second.
That's not a usable interactive worker. It's barely a usable batch worker. And nothing about
the capacity check hinted at it.
The arithmetic that would have told me in advance
Autoregressive generation reads the entire model's weights once per token. So:
ceiling ≈ memory bandwidth ÷ bytes touched per operation
= 120 GB/s ÷ 15 GB
= 8 tokens/second
The two Stop hooks behind this note are on GitHub under MIT: hexisteme/hard-gate-hooks. They ship with their tests and a read-only scanner that prints what they did on **your* machine, not mine — including the case where it tells you they aren't worth wiring up yet. No email, no signup.*
Measured 5.6 against a ceiling of 8. Ratio 0.70.
That ratio is the whole verdict. When measured throughput is a large fraction of the arithmetic
ceiling, you are bandwidth-bound, and you now know something concrete: the bottleneck is
not your configuration, not memory pressure, not thermal throttling. It's how fast bytes move.
Rule of thumb I now use: ratio ≥ 0.5 → bandwidth-bound, and size-reduction fixes are dead.
Why "just quantize harder" doesn't work
The natural move when capacity is tight is to shrink. Lower quantization, smaller batch,
heavier compression. It's the reflex, and in a bandwidth-bound regime it's close to useless.
I was considering Q3_K_M at 13.8GB. Run the same division:
120 ÷ 13.8 = 8.7 tokens/second (up from 8)
Under 9% more throughput. For a real drop in output quality, because quantization error
doesn't scale linearly with size the way bandwidth does — you give up more than you get, every
time, in this regime.
I killed that plan without downloading anything. That's the saving this rule buys: an
arithmetic rejection instead of an afternoon of benchmarking a model that couldn't have won.
The deeper reason is that capacity and throughput are governed by different resources.
Capacity is bytes of memory. Throughput is bytes per second across a bus. Pulling the capacity
lever moves the capacity number. It touches the throughput ceiling only through the incidental
fact that a smaller model has fewer bytes to stream — a weak, strictly linear coupling, and you
pay for it non-linearly.
Naming the right bottleneck tells you which lever works
This is the part that makes the arithmetic worth doing. Once you know it's bandwidth, the same
model on different hardware is a division away:
| Machine | Bandwidth | Ceiling for a 15GB model |
|---|---|---|
| M4 | ~120 GB/s | 8 tok/s |
| M4 Pro | ~273 GB/s | 18 tok/s |
| M4 Max | ~546 GB/s | 36 tok/s |
The verdict flips on hardware, not on model size. "This model is too slow" was never true —
"this model is too slow on 120GB/s" was. Those lead to completely different purchase
decisions, and only one of them is correct.
When you correctly name the constrained resource, the set of interventions that can possibly
work falls out of it. Naming the wrong one sends you tuning things that were never the
problem.
Anti-patterns, all of which I've done
- "It loaded, so we can use it." Loading is the capacity gate. Passing it leaves throughput entirely unknown.
- "It's slow, so quantize lower." In a bandwidth-bound regime, size and speed are linearly coupled and quality degrades faster. This is the classic symptom of conflating the two gates.
- "It must be swap / memory pressure." Measure warm, at least once, with the model already resident. Otherwise you're mixing the ceiling with transient congestion — and if you go clean up applications before separating them, you'll never find the actual cause. Here there was no pressure at all and the ceiling was exactly where the arithmetic put it.
- "Theoretical bandwidth isn't effective bandwidth, so the calculation is meaningless." Effective is typically 60–80% of theoretical. Include that and the estimate is still good to an order of magnitude — and order of magnitude is the entire decision. 8 vs 5.6 is the same answer. 8 vs 36 is a different one.
The procedure
-
Before checking capacity, compute the throughput ceiling:
ceiling ≈ bandwidth ÷ bytes touched per operation. - Measure at least once warm, so cold-load and swap pressure don't contaminate the number.
- If
measured / ceiling ≥ 0.5, declare bandwidth-bound and reject size-reduction fixes. - Choose from what's actually left: (a) restrict to latency-tolerant uses, (b) change hardware, (c) move the work off this machine.
- Write the falsifier as a bandwidth number. If you've named the constrained resource, the point at which that resource changes is the condition that overturns your verdict.
That last step is why this generalizes past local models. The same shape applies to any
streaming bottleneck — disk-bound ETL, network-bound sync, cache-line-bound inner loops.
"It fits" and "it's fast enough" are separate gates, and the first one is free to check, which
is exactly why it gets mistaken for the second.
What would change my mind
Bandwidth is the named bottleneck, so the falsifier is a bandwidth number: on a 273GB/s
machine this model clears 18 tok/s and the "not usable as a live worker" verdict is void. Any
architecture change that stops reading all weights per token — heavy MoE sparsity, aggressive
speculative decoding — also breaks the bytes touched = model size assumption the ceiling is
built on, and the division has to be redone with the real figure.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (2)
The bandwidth-÷-model-size ceiling is the back-of-envelope everyone should run before downloading anything, and the 0.70 ratio verdict is a great way to make it concrete. The "quantize harder is dead in this regime" conclusion is the counterintuitive bit worth shouting — people reach for smaller quants expecting throughput and get quality loss for a rounding error of speedup.
The one escape hatch from the "reads all weights once per token" assumption is batching and MoE: with concurrent requests you amortize the weight read across multiple tokens in flight, so the per-token ceiling stops being your throughput ceiling — a bandwidth-bound single stream can still saturate compute at batch size 16. And MoE flips the arithmetic because you only touch the active experts per token, not the full parameter count, which is why a 27B dense and a 30B-A3B feel completely different on the same box. Did you get a chance to measure how the ratio moves once you push concurrent generations through that M4?
No, I haven't measured concurrency on that box, so the honest answer is the arithmetic plus the falsifier rather than a number.
Both escape hatches are real, and batching is a genuine gap in the post. The closing section names MoE and speculative decoding as things that break
bytes touched = model sizeand says nothing about amortizing the read across streams. It should have.Where batching stops helping is the same division applied once more, to the machine instead of the model. A dense decode step at batch B costs about 2 × params × B FLOPs and still reads the weights once, so the step stays bandwidth-bound until
2·B / bytes_per_paramcrosses the machine's ridge,peak FLOPS ÷ bandwidth. Apple doesn't publish a TFLOPS figure for the M4 GPU; treating it as a ~4 TFLOPS class part (roughly double that if fp16 runs at twice the fp32 rate) gives a ridge of ~33–67 FLOP/byte. IQ4_XS is ~0.55 bytes/param, so intensity is ~3.6·B and the knee lands at B ≈ 9–18. Past that the step is compute-bound, and the aggregate ceiling tops out somewhere near 9–18 × 8 tok/s before the KV-cache reads that scale with B × context pull it back down. So "batch 16 saturates compute" is right in direction and, on this box, arrives early: 16 streams are at or past the knee. On an H100 the same ridge is ~300 FLOP/byte and 16 streams are nowhere near it, which is why the batching intuition depends on where you formed it.Two caveats on MoE for a 24GB machine specifically. First, the capacity gate comes back: 30B-A3B at 4-bit is 16.5–18.5 GB depending on the quant, against a 17.76 GB
recommendedMaxWorkingSet. The model that flips the throughput arithmetic re-tightens the gate the post called free. Second, active-parameter count understates the traffic: shared weights and the router are read every token regardless, and at batch B the union of experts selected across the batch grows toward the whole model, so bytes touched per step climbs back toward the dense figure. Batching and MoE partly cancel; each hatch works best alone.Falsifier in the post's own currency: if 8 concurrent streams on the M4 deliver materially more than ~18× the single-stream rate, the FLOPS assumption is wrong and the knee is further out. That measurement is what this reply is missing, with two conditions on it: the runtime has to actually batch the decode (Ollama only does with
OLLAMA_NUM_PARALLELabove 1, which on a 24GB box it may not pick by default), and aggregate tokens/s and per-request latency have to be reported separately, because batching raises the first and never the second. The single-stream 8 tok/s is still the number a live worker feels.