Credit: Solen Feyissa on Unsplash DeepSeek has open-sourced DeepInfer, the software it uses to run its AI models in production. The Chinese AI lab announced the release on X on Tuesday and published the code on GitHub under the Apache 2.0 licence. For now, it supports only DeepSeek-V4.1-Flash. \ud83d\ude80 Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency.<\/p>\n
We\u2019re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable. \ud83d\udd17 https:\/\/t.co\/JGAik1kgk0 1\/4 \u2014 DeepSeek (@deepseek_ai) October 20, 2026 Inference is the work of running a trained model to answer requests. The same model can cost much more or less to run, depending on the software that serves it. What is in the release DeepInfer bundles optimised code for DeepSeek\u2019s models with ways to split the work across many chips. It also includes speculative decoding, a technique that drafts several tokens ahead and checks them in one pass.<\/p>\n
Other features keep outputs consistent from run to run and force answers into set formats, according to the GitHub page. It runs on two kinds of hardware: Nvidia\u2019s SM100 data-centre GPUs and Huawei\u2019s Ascend 950 chips. DeepSeek said code for future models and further improvements will land in the same place. The release is not a complete service on its own.<\/p>\n
It leaves out a cache system, the split between processing prompts and generating answers, and load balancing across machines, the GitHub page says. DeepSeek also released deepseek-recipe, which matches how its official API turns text into tokens and back. Built on vLLM DeepInfer was first built mainly on vLLM, a widely used open-source inference engine, DeepSeek says. It also drew on SGLang and Nvidia\u2019s TensorRT-LLM.<\/p>\n
The vLLM project thanked DeepSeek on X for crediting the open-source community. It said it would keep improving its own support for V4.1-Flash with ideas from DeepInfer. DeepSeek measured its published benchmarks on four SM100 GPUs for processing prompts and 64 for generating output. They are the company\u2019s own figures.<\/p>\n
There is no independent benchmarking of DeepInfer\u2019s gains yet, Bloomberg\u2019s Saritha Rai reported. Why it matters for cost Cloud and hosting companies that already run DeepSeek\u2019s models can now use the same methods DeepSeek uses. Companies running the models on their own servers could also cut costs, Bloomberg reported. AI companies usually guard these techniques closely, Bloomberg noted.<\/p>\n
The release follows DeepSeek\u2019s September model launch of V4.1-Flash. Earlier this month, it published Huawei chip tools as an alternative to Nvidia\u2019s CUDA. Open-weight models now take 23% of spending on Vercel\u2019s AI Gateway. It comes a day after the US ordered civilian federal agencies to remove DeepSeek products from their systems.<\/p>\n
By Tuesday afternoon, the GitHub repository had more than 8,000 stars. Get the TNW newsletter \u2715 Close Get the TNW newsletter Get the most important tech news in your inbox each week.<\/p>\n","protected":false},"excerpt":{"rendered":"
Credit: Solen Feyissa on Unsplash DeepSeek has open-sourced DeepInfer, the software it uses to run its AI models in production. The Chinese AI lab announced the release on X on Tuesday and published the code\u2026<\/p>\n","protected":false},"author":1,"featured_media":29501,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-29500","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/demo.tcjhomeaccents.com\/wp\/wp-json\/wp\/v2\/posts\/29500","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/demo.tcjhomeaccents.com\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/demo.tcjhomeaccents.com\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/demo.tcjhomeaccents.com\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/demo.tcjhomeaccents.com\/wp\/wp-json\/wp\/v2\/comments?post=29500"}],"version-history":[{"count":0,"href":"https:\/\/demo.tcjhomeaccents.com\/wp\/wp-json\/wp\/v2\/posts\/29500\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/demo.tcjhomeaccents.com\/wp\/wp-json\/wp\/v2\/media\/29501"}],"wp:attachment":[{"href":"https:\/\/demo.tcjhomeaccents.com\/wp\/wp-json\/wp\/v2\/media?parent=29500"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/demo.tcjhomeaccents.com\/wp\/wp-json\/wp\/v2\/categories?post=29500"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/demo.tcjhomeaccents.com\/wp\/wp-json\/wp\/v2\/tags?post=29500"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}