DeepSeek DeepInfer Won't Run on Your RTX 5090. Here's What to Run Instead

DeepSeek open-sourced DeepInfer, the inference engine behind its own production services, under Apache 2.0. The HN thread passed 2,200 points, and a lot of the excitement I've seen boils down to "finally, fast DeepSeek on my own box."

That isn't going to happen for most of us. Officially, DeepInfer runs on NVIDIA SM100 (B200/GB200) or Huawei Ascend 950, not on an RTX 5090, it serves exactly one 763B model, and it has no HTTP server. You run a Python script and it generates tokens.

I still think the release is worth reading. But the gap between what people think they got and what the README actually describes is big enough to go through point by point.

DeepInfer hardware and software requirements

Straight from the README:

• One model: DeepSeek-V4.1-Flash. The default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017.
• NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs.
• Linux, GCC 13.0+, Python 3.12, CUDA 13 (or CANN 9.2 on Ascend).
• Offline inference only.

SM100 means datacenter Blackwell, the B200/GB200 class. Your RTX 5090 is also Blackwell, but it reports compute capability 12.0 (SM120). That's a different target, and the README doesn't list it.

The model is the other wall. The Hugging Face card lists 763B parameters, stored as a mix of I8, FP8, BF16 and F32 tensors. The repo comes to about 510 GB of weights, before you allocate anything for activations. Despite the "Flash" in the name, this is not a small model.

The quickstart shows what kind of tool this is:

# fresh Python 3.12 env
export DEEPINFER_ACCELERATOR=nvidia   # or ascend
./build.sh
python examples/offline_inference.py

There's no deepinfer serve and no OpenAI-compatible endpoint.

One caveat to "you can't run it": someone in the HN thread already linked a community port to older hardware, HalfVulpes/Deepinfer-SM80. It's unofficial, I haven't tested it, and it doesn't make the weights any smaller.

No KV cache system, no server: what DeepSeek left out

The README is upfront about what this release doesn't include, and it's mostly the stuff that turns an engine core into a service:

• a KV cache system
• VE/prefill/decode disaggregation (the README says most of the engine work for it is already done)
• load balancing across data-parallel replicas for prefill and decode

You need those pieces before you can put real traffic on a set of fast kernels. Without a KV cache system there's no prefix caching across requests, and prefix caching is a big reason DeepSeek's own API is so cheap on repeated prompts. As HN commenter mobelkh put it, "Deepseek manages to be ridiculously cheap and efficient with prompt caching." That piece didn't ship.

DeepInfer vs vLLM: no benchmark to compare

This is my main complaint. The README has prefill and decode Pareto charts, measured on 4 SM100 GPUs for prefill and 64 SM100 GPUs for decode, each set inside one scale-up domain. The page has no table of numbers, and it doesn't compare against vLLM, SGLang or TensorRT-LLM on the same hardware.

I want that baseline because the README says DeepInfer was built primarily on vLLM, with ideas from SGLang and TensorRT-LLM. Without it you can't tell how much of the speed is DeepSeek's engineering and how much is 64 Blackwell GPUs.

Tepix asked the obvious question on HN: "It's a project based on vLLM, explain why you consider it a total game changer?" In the same subthread cmrdporcupine pointed out that vLLM and SGLang can already serve V4.1 Flash, "less efficiently, I'm sure, but on much broader hardware," and that "a lot of what's in here is already in published papers out of DeepSeek already."

If you have the rack, you can reproduce the sweep with examples/sweep/run_sweep.sh. To put a price on that: one HN commenter put B200 cloud rental at "~$6-14/hour". Across the 64-GPU decode setup that comes to about $384–896 per hour.

The eval README gives quality numbers for V4.1 at default settings: AIME 2026 99.9%, GPQA Diamond 93.4%, HLE-Full 42.6% (Pass@1). So the engine gives correct answers, which tells you nothing about how fast it is next to vLLM.

The parts worth reading

There's real engineering in here, and it's now Apache 2.0 code you can read and port.

Start with the MoE side. DeepInfer does expert parallelism with expert-parallel load balancing (EPLB), alongside attention data parallelism and balanced context parallelism. The kernels come from DeepSeek's earlier open-source releases (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect) plus custom TileLang kernels. If you work on MoE code paths in vLLM or SGLang, you get to see how DeepSeek fits the pieces together in production.

Then there's batch-invariant, deterministic sampling. DEEPINFER_BATCH_INVARIANT=1 makes outputs independent of whatever else is in the batch. Turn that on, set DEEPINFER_ENABLE_MTP_VARLEN=0, pass a fixed seq_seed in SamplingParams, and the same input gives the same tokens. NVIDIA only. vLLM has its own version (VLLM_BATCH_INVARIANT=1), so the feature itself isn't new. I find it more interesting that DeepSeek runs it in a production engine. If you've ever chased an eval regression that turned out to be batch-composition noise, you'll want this.

Speculative decoding goes through DSpark. The engine drafts tokens, and confidence estimates decide how verification gets scheduled. You enable it with DEEPINFER_ENABLE_MTP_VARLEN=1, and DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length (default 5). Because deterministic mode requires MTP varlen off, you have to choose between the two.

The engine also supports structured output (regex, JSON schema or EBNF via so_args), turns on async scheduling by default, and uses SWA bounded replay on the decoder to cut prefill work.

How to run DeepSeek V4.1-Flash locally instead

If you want V4.1-Flash on your own hardware today, the Hugging Face model page lists the options. You get vLLM and SGLang (both serve OpenAI-compatible APIs), plus quantized builds for llama.cpp, Ollama and LM Studio. DeepInfer isn't on the list.

Run this before you try anything:

nvidia-smi --query-gpu=name,compute_cap --format=csv

10.0 means you have the hardware the README targets. If you see anything else, use vLLM or SGLang. B300 reports 10.3, and the README doesn't say whether that works. If you get 12.0 you're on consumer Blackwell, and for a 763B model your realistic option is a quantized GGUF in llama.cpp, with a lot of system RAM. Unsloth's guide puts its 1-bit quant at 128GB+ and builds llama.cpp from a PR branch, not a stock release.

What it means for vLLM and SGLang

I think the best outcome is vLLM and SGLang maintainers reading the EPLB and DSpark code and upstreaming whatever generalizes, whether or not anyone ever runs DeepInfer itself. mobelkh hoped for the same thing, for "the ideas here being incorporated in other Opensource projects." The license allows it, and since DeepInfer started from vLLM, porting back should be easier than usual.

The Ascend support also hints at where DeepSeek is headed: its production engine treats Ascend 950 as a first-class target next to SM100. Or, as mordae put it on HN: "Global south will be buying Ascends soon."

My take: this is a good reference implementation that got hyped as a product. Read the code, but don't plan on running it.

I keep going back and forth on this one. If an inference engine officially runs only on a rack of datacenter Blackwell, is it "open" in a way that matters to you? Or is code you can read and fork enough, even if you never execute it?

Sources

• deepseek-ai/DeepInfer README
• DeepInfer eval README
• DeepSeek-V4.1-Flash-1017 model card
• Hacker News discussion
• vLLM batch invariance docs
• Unsloth: DeepSeek-V4.1-Flash GGUF guide