DEV Community

Shivam Kumar
Shivam Kumar

Posted on

DeepSeek Just Open-Sourced Its Production Inference Stack — Here's What I'd Steal for My Own Serving Setup

#llm #ai #performance #opensource

DeepSeek Just Open-Sourced Its Production Inference Stack — Here's What I'd Steal for My Own Serving Setup

Research note — VisionQuantech, October 22, 2026

On October 20, DeepSeek published DeepInfer on GitHub under Apache 2.0 — the actual inference framework it says powers its large-scale production services, targeting DeepSeek-V4.1-Flash. Not a paper. The working stack. Here's the builder's read: what it is, what to take from it, and what its limits teach you about serving LLMs at scale.

TL;DR

  • What: production inference engine for DeepSeek-V4.1-Flash, Apache 2.0
  • Hardware: NVIDIA SM100 (Blackwell) + Huawei Ascend 950 — first-class support for both in one codebase
  • Mode: offline inference only; no KV-cache system, no prefill/decode disaggregation, no replica load balancing
  • Signature features: DSpark speculative decoding with confidence-scheduled verification, async zero-overhead scheduling, batch-invariant deterministic inference, structured outputs (regex/JSON schema/EBNF)
  • Production numbers (DeepSeek's, unverified): 155 trillion tokens/day per cluster, 96% input cache hit rate, ~400ms TTFT, 230 tok/s output (300 for code), ~$38/GPU-hour revenue at valley pricing

The quick start that matters

Build requirements: Linux, GCC 13+, Python 3.12, CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 downloads into the Hugging Face cache on first run. The shape of a first run, from the README's documented flow:

git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer

# empty Python 3.12 environment first, per the README
export DEEPINFER_ACCELERATOR=nvidia   # or: ascend (Huawei Ascend 950)
./build.sh

# runs the default checkpoint, downloaded to the Hugging Face cache on first run
python examples/offline_inference.py

# or point at a local copy of the weights
DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 python examples/offline_inference.py
Enter fullscreen mode Exit fullscreen mode

Two things to notice. First, structured outputs are a first-class citizen — regex, JSON schema, or EBNF grammar constraints built into the sampler, not bolted on with a wrapper. If your pipeline parses LLM output (and whose doesn't), grammar-constrained generation is the single highest-ROI feature to lift from this release into your own stack.

Second, deterministic sampling + batch-invariant inference: identical inputs produce identical outputs regardless of batch composition. If you've ever debugged a flaky eval that only fails under load, you know why this exists. It's currently NVIDIA-only — porting the batch-invariant path is one of the obvious community contributions this release invites.

For the reference evals, the repo ships an eval folder with driver scripts (bash eval/aime/run.sh); datasets aren't downloaded for you, HLE and GPQA are gated on Hugging Face, and scoring defaults to an LLM judge service (set the EVAL_JUDGE_* variables or use the rule-based judge for AIME/GPQA). Security note before you build: this compiles vendored operators and pulls multi-gigabyte weights — use a disposable machine or container, and read build.sh first.

DSpark: speculative decoding, done the production way

The feature I'd study first. Speculative decoding drafts several tokens ahead and verifies them in one pass. DeepInfer's DSpark adds confidence-scheduled verification: the draft step emits tokens plus confidence estimates, and the verifier uses those estimates to schedule what gets checked. Default maximum draft length is 5.

Why this is the production-correct version of an old trick: naive speculative decoding wastes compute verifying drafts that were obviously going to be rejected. Confidence-scheduled verification skips the doomed checks. This is the kind of optimization that only survives contact with real traffic — and it's exactly the kind of optimization frontier labs never used to share.

The architecture checklist, as a syllabus

The README's component list is worth treating as a checklist for any serious inference setup:

  1. Kernels: FlashMLA (attention), DeepGEMM, DeepEP (MoE communication), TileKernels, DeepSelect, custom TileLang kernels. The MoE kernels are the load-bearing ones — V4.1-Flash is an MoE model, and expert-parallelism efficiency is what separates a cheap serve from an expensive one.
  2. Parallelism: attention data-parallel with balanced context parallelism, MoE expert-parallel with EPLB (expert load balancing). If you're serving any MoE model, EPLB is the mechanism to study.
  3. SWA bounded replay for long-context efficiency.
  4. Batch-invariant + deterministic modes for reproducible evals.

And the lineage credit: "originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM." Reading between the lines, this is a map of which open-source serving ideas survived production hardening. vLLM won the architecture lottery here.

What the release deliberately excludes — and why that's a lesson

DeepInfer ships without a KV-cache system, without prefill/decode disaggregation, without load balancing across data-parallel replicas. The README says so plainly. That's not an omission — it's the honest boundary between "inference engine" (what you got) and "serving stack" (what DeepSeek runs). Benchmarks were measured on 4 SM100 GPUs for prefill and 64 for decode, per MarkTechPost's reading.

The lesson for builders: serving at 155T tokens/day is 20% inference engine, 80% everything around it — caching, disaggregation, scheduling policy, fleet management. Don't mistake the open-sourced 20% for the whole machine. But do treat it as the first time anyone has shown you the 20% in complete working form.

The dual-silicon bet

The headline the press underplayed: one codebase, NVIDIA SM100 and Huawei Ascend 950, same first-class support. DeepSeek said last week it plans to pre-train on Huawei silicon by end of 2026. A unified serving framework is what makes that plan executable — same model, same serving code, whichever chips you can buy. In an export-control world, hardware-agnostic serving is infrastructure strategy, not generosity.

The repo reportedly crossed 8,000 GitHub stars within hours — a day after the US DHS banned DeepSeek from federal agencies. Code doesn't clear customs.

What to actually do with this

  1. Clone it and read the scheduler. The async batch scheduler and DSpark verification logic are the highest-value code in the repo.
  2. Port batch-invariant inference to your stack if you run evals under load.
  3. Adopt grammar-constrained generation (regex/JSON schema/EBNF) as a default pattern in your pipelines.
  4. Don't try to rebuild DeepSeek's economics from this repo alone — the 96% cache hit rate comes from their traffic, not their code.

The gap between a systems paper and a production stack is where ML infra teams spend their years. DeepSeek just handed you a worked example of how to cross it. Read it like a syllabus.

References

Published: Dev.to, Hashnode, Medium, Archive.org — October 22, 2026.

Top comments (0)