Speculative Decoding vs Distillation: Two Ways to Buy Latency, One of Them Charges Quality

A user typing into a chat box will tolerate roughly two hundred milliseconds between tokens before the interface feels broken. A 70B model on a single accelerator produces a token every forty milliseconds on a good day, and every hundred on a bad one. The gap between what the hardware does and what the user expects is where two very different techniques compete.

One borrows compute from a small model to draft tokens. The other replaces the model with a smaller one trained to imitate it. They are often listed as interchangeable optimisations. They are not, and the difference shows up on the invoice and in the support queue.

Part One: Drafting is a bet, not a compression

Speculative decoding runs a small draft model ahead of the large one. The large model then verifies the draft in a single forward pass, accepting the tokens it agrees with and rejecting the rest.

The economics depend entirely on the acceptance rate. If the draft is accepted ninety percent of the time, one large-model pass yields several tokens, and the wall clock per token falls sharply. If acceptance drops to fifty percent, you have paid for the draft and the verification and gained almost nothing.

Acceptance is high when the draft model has seen the same distribution — same tokenizer, same training corpus, same domain. It collapses when the task is unusual, which is precisely when users are most likely to be watching the screen.

The critical property is that speculative decoding is lossless. The output distribution is identical to the large model's, because the verification step is exact. You are trading compute for latency, and the compute is free only if the accelerator had spare capacity.

Part Two: Distillation is a compression, and it charges quality

Distillation trains a small model to reproduce a large model's outputs. The result is a genuinely smaller model: fewer parameters, less memory, lower cost per token regardless of spare capacity.

The price is that the model is now a different model. It is usually good at the distribution it was trained on and worse at everything else. Benchmarks move a few points. The tail moves a lot.

There is a subtler cost. Distillation bakes in a snapshot of the teacher's behaviour. When the teacher is updated, the student is stale, and re-distilling is a training run, not a config change. Teams that ship distilled models discover they have taken on a recurring training obligation they did not budget for.

Distillation buys permanent cost savings by giving up the ability to change the model without retraining it; speculative decoding buys temporary latency savings and gives up nothing but idle compute.

Part Three: The rack decides which one wins

Here is the part that most comparisons miss. Speculative decoding consumes extra FLOPs. If your rack is at thirty percent utilisation, those FLOPs are free and the technique is close to a pure win. If your rack is at eighty-five percent, the draft model is competing for the same silicon as paying traffic, and the effective cost per token rises.

Distillation does not care about utilisation. A smaller model is cheaper per token whether the rack is idle or full. That makes it the correct choice for a cost-constrained deployment running hot, and the wrong choice for a latency-constrained deployment with headroom.

The two compose, and the composition is usually the right answer: distil to a smaller model, then speculate within it using an even smaller draft. The quality loss is set by the distillation step, and the latency gain is set by the draft acceptance rate.

Part Four: What to measure

Measure acceptance rate on production traffic, not on a benchmark set. Draft models look excellent on in-distribution text and mediocre on the odd queries that dominate real usage.

Measure time to first token separately from inter-token latency. Speculative decoding improves the second and can worsen the first, because the draft adds a serial step before the first verified token.

Measure quality on the tail, with a rubric, not with a benchmark average. A distilled model's failures are concentrated in the cases your benchmark does not contain.

Measure the retraining obligation. If the teacher changes monthly, distillation is not a one-off project; it is a pipeline with an owner.

What is actually transferable

The pattern is general: there are only two ways to make a system faster, borrow resources that are idle or remove work that is being done, and only the first is reversible.

Caching borrows idle storage. Precomputation borrows idle time. Speculative decoding borrows idle compute. All three are reversible — turn them off and the system returns to its previous behaviour. Compression, distillation, and approximation remove work, and they cannot be turned off without going back to the original.

That distinction should drive the order of adoption. Exhaust the reversible options first, because they cost nothing but engineering, and reach for the irreversible ones only when the reversible options are spent. Most teams do the opposite, because compression is easier to demo.

SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.