Skip to content

DEV Community

Serving at Scale Series' Articles

Back to AI Tech News's Series

Cover image for What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You

Oct 6

What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You

#llm #vllm #performance #devops

6 min read

Oct 6

KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly

#llm #vllm #performance #devops

7 min read

Oct 12

On-Device LLM Inference: The Tradeoffs Nobody Puts on the Spec Sheet

#ai #machinelearning #llm #edgeai

3 min read

Cover image for Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off

Oct 19

Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off

#ai #machinelearning #llm #vllm

11 min read

Cover image for Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax

Oct 20

Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax

#ai #machinelearning #vllm #performance

12 min read

Cover image for KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off

Oct 21

KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off

#ai #machinelearning #vllm #llm

14 min read

Cover image for One Fleet for Serving and Training? What a 325M Pretraining Run on an Inference NPU Means for AI Infrastructure

Oct 26

One Fleet for Serving and Training? What a 325M Pretraining Run on an Inference NPU Means for AI Infrastructure

#ai #machinelearning #infrastructure #hardware

2 min read