DeepInfer
A high-performance, large-scale large-model inference framework open-sourced by DeepSeek, aimed at deployment and inference optimization for DeepSeek models. It involves engineering optimizations such as KV Cache management and batching strategies.
Core Facts
DeepSeek is developed by DeepSeek (深度求索) company.
In its July changelog, DeepSeek credited its internal Harness (the "minimal mode") as the behind-the-scenes contributor to V4 Flash's programming benchmark scores.
DeepInfer is an open-source state-of-the-art (SOTA) inference system.
Timeline (last 90 days)
Some users in overseas developer communities called DeepInfer "the real OpenAI."
What DeepInfer open-sources is the inference stack used in production environments, not the model weights.
DeepSeek open-sourced the inference framework DeepInfer on October 20, aiming to reduce the compute cost and energy consumption of model inference.
The open-source version of DeepInfer is not a complete production system and does not include parts coupled with internal infrastructure, such as KV buffer management, cross-machine prefill-decode disaggregation, and load balancing across replicas.
DeepInfer currently supports only one model, V4.1 Flash, and does not support Hopper and earlier GPUs.
The open-source version of DeepInfer supports only NVIDIA SM100 and Huawei Ascend 950 hardware, and only supports offline inference.
The official recommended starting configuration for running DeepInfer efficiently is 64 cards for prefill plus 6 tokens.
DeepInfer splits inference into two stages, prefill and decode, and uses disaggregated deployment: prefill uses 8-card machines, decode uses supernodes of 64 to 128 cards, and the card requirements of the two are roughly 1 to 1.
DeepInfer is released under the Apache 2.0 license, supports NVIDIA Blackwell and Huawei Ascend 950, and includes components such as expert load balancing, speculative decoding, and asynchronous scheduling.
DeepInfer's core mechanism is context caching, which keeps already-computed mathematical paths in memory, with a cache validity period exceeding 72 hours.
34 more timeline events
All Facts (20)
DeepSeek is developed by DeepSeek (深度求索) company.
80%VerifiedIn its July changelog, DeepSeek credited its internal Harness (the "minimal mode") as the behind-the-scenes contributor to V4 Flash's programming benchmark scores.
65%Partially VerifiedDeepInfer is an open-source state-of-the-art (SOTA) inference system.
70%Partially VerifiedDeepSeek open-sourced the inference framework DeepInfer on October 20, providing optimized operators, multi-card parallelism and asynchronous scheduling capabilities for V4.1 Flash, under the Apache 2.0 license.
70%UnverifiedWhat DeepInfer open-sourced is the offline inference core; the complete online service chain still requires filling in the caching layer and load balancing yourself.
60%UnverifiedAccording to the video information, DeepInfer is compatible with both the NVIDIA and Huawei Ascend platforms, a single cluster can reach a daily throughput of 155 trillion tokens, and each GPU can earn $38 per hour.
60%UnverifiedWhat DeepInfer open-sources is the inference stack used in production environments, not the model weights.
50%UnverifiedSome users in overseas developer communities called DeepInfer "the real OpenAI."
50%UnverifiedDeepInfer currently supports only one model, V4.1 Flash, and does not support Hopper and earlier GPUs.
50%UnverifiedThe open-source version of DeepInfer is not a complete production system and does not include parts coupled with internal infrastructure, such as KV buffer management, cross-machine prefill-decode disaggregation, and load balancing across replicas.
50%UnverifiedThe open-source version of DeepInfer supports only NVIDIA SM100 and Huawei Ascend 950 hardware, and only supports offline inference.
50%UnverifiedDeepInfer splits inference into two stages, prefill and decode, and uses disaggregated deployment: prefill uses 8-card machines, decode uses supernodes of 64 to 128 cards, and the card requirements of the two are roughly 1 to 1.
50%UnverifiedThe official recommended starting configuration for running DeepInfer efficiently is 64 cards for prefill plus 6 tokens.
50%UnverifiedDeepInfer is released under the Apache 2.0 license, supports NVIDIA Blackwell and Huawei Ascend 950, and includes components such as expert load balancing, speculative decoding, and asynchronous scheduling.
50%UnverifiedDeepInfer's core mechanism is context caching, which keeps already-computed mathematical paths in memory, with a cache validity period exceeding 72 hours.
50%UnverifiedDSpark in DeepInfer first estimates the probability that a draft will pass, then, combined with hardware throughput, adjusts how long a draft each request is worth verifying.
50%UnverifiedThis episode's content is compiled from official documentation, and no deployment speed tests were conducted.
50%UnverifiedOfficial DeepInfer data shows that a single cluster can reach a daily throughput of 155 trillion Tokens, an input cache hit rate as high as 96%, and chat caches retained for up to seven days.
50%UnverifiedDeepInfer only supports the V4.1-Flash model.
50%UnverifiedEnterprises using DeepInfer still need to build their own service-layer components such as gateways, authentication, monitoring, and canary releases.
50%