DeepInfer Open-Sourced: A Deep Dive into DeepSig's Production-Grade Inference Engine

DeepSig has open-sourced DeepInfer, the high-performance inference engine powering its large-scale production services, under the Apache 2.0 license—a rarity in the inference framework space, where most vendors release only model weights while keeping the engineering details that truly determine cost and latency under wraps. The public release of DeepInfer's code means the community can, for the first time, systematically examine how a production engine handling 155 trillion tokens per day is built.

The Technical Foundation of a Production Engine

DeepInfer is not built from scratch; it is built on top of vLLM with custom optimized kernels layered on top. The current version focuses on offline inference, supports the DeepSig V4.1 Flash model, and is compatible with NVIDIA SM10 GPUs and Huawei Ascend 950 NPUs at the hardware level. This dual-hardware support gives it some promise for adapting to domestic compute platforms.

Custom Kernels and Key Mechanisms

The core components mentioned in public materials include three types of custom optimized kernels: Flash MLA, Deep GMM, and Deep EP. Around these kernels, DeepInfer also integrates mechanisms such as DeepSpark speculative decoding with consistency scheduling verification, SWA bounded replay, asynchronous zero-overhead scheduling, and structured output.

Judging from the naming and the combination of features, the design centers on "scheduling overhead" and "verification cost": speculative decoding boosts throughput only if the verification step is lightweight enough, while asynchronous scheduling attempts to move scheduling itself off the critical path. These details have rarely appeared in full in open-source implementations of production engines.

Speculative Decoding is a technique that accelerates large model generation without changing the output distribution: a lightweight small model (or multiple heads of the same model) quickly drafts several candidate tokens, then the large model verifies these candidates in parallel in one pass, accepting the segments that match the original distribution and resampling the rejected parts. Its throughput gain depends heavily on whether the "verification cost" is low enough—if verification itself is expensive, the speedup from drafting is offset. DeepInfer pairs it with "consistency scheduling verification" precisely to control verification overhead at the scheduling level, so that speculative decoding can still deliver its speedup ratio under real workloads.

SWA Bounded Replay (Sliding Window Attention, bounded replay) refers to replaying or retaining history context outside the window with a bounded length under the sliding window attention mechanism, in order to maintain long-context consistency within limited VRAM and compute budgets. It directly affects the usability of long conversations and multi-turn scenarios, and is also one of the prerequisites for achieving a high cache hit rate—only with a clear strategy for window management will prefix reuse not fail due to context truncation.

Production Data Behind 155 Trillion Tokens

The production figures given in the video are key to understanding DeepInfer's positioning: a single cluster can process 155 trillion tokens per day, with a cache hit rate of 96%.

The 96% cache hit rate deserves separate discussion. In real services, high cache hit rates usually come from prefix reuse, shared system prompts, and context continuation in multi-turn conversations. This means DeepInfer's design clearly leans toward production workloads with highly repetitive context, rather than one-off, low-repetition request scenarios. For teams building RAG, Agent, or long-conversation products, these workload characteristics are highly informative.

Prefix Caching and cache hit rate are key to understanding the 96% figure. In inference services, if multiple requests share the same prefix (such as system prompts, few-shot examples, or the generated history in multi-turn conversations), the engine can reuse the KV cache corresponding to that prefix and skip repeated prefill computation. The higher the hit rate, the more token computation is saved, which directly translates into lower latency and GPU-hour savings. A 96% hit rate indicates that DeepInfer faces workloads with highly repetitive prefixes, and that its cache eviction and indexing strategies are stable enough at production scale; for open-domain scenarios where each request has a different prefix, this number would typically drop significantly.

The Real Significance of Open Sourcing: Exposing the Cost Structure

According to the video author, the major significance of open-sourcing DeepInfer is that it lets the community see the internal implementation of a production-grade inference engine, thereby helping reduce compute costs, cut energy consumption, and make AI more affordable.

This judgment has merit. The bulk of inference cost often lies not in the model itself, but in the "invisible places"—scheduling strategies, cache reuse, and kernel efficiency. When these implementations are made public, small and medium teams do not have to repeat the trial and error; they can directly borrow the scheduling and caching ideas to compress the cost curve of building their own inference services. For an engine handling trillions of tokens per day, even a few percentage points of efficiency improvement translate into substantial numbers in electricity bills and GPU hours.

The Limitations Are Equally Clear: It's an Offline Engine

It is important to view this calmly: the current version of DeepInfer is an offline engine and does not provide out-of-the-box online serving capabilities. The video explicitly points out that it lacks a complete KV cache system, prefill/decode separation, and load balancing.

These three are precisely the core pieces of an online inference service. The lack of a complete KV cache system means limited cross-request context reuse; the lack of prefill/decode separation makes it difficult to optimize resource allocation separately for the two types of workloads; and the absence of load balancing makes it hard to directly handle online traffic across multiple replicas and instances. In other words, DeepInfer is more like a high-quality "reference implementation of inference kernels and scheduling" than a finished product that can directly replace vLLM in production deployment.

For teams that want to go live directly, this means they still need to build the serving layer themselves; for engineers who want to study the internals of a production engine, this is exactly where its value lies.

Prefill/Decode Separation (PD separation) is a key architecture for online inference services: the prefill phase processes input context, is compute-intensive, and exhibits large-batch characteristics; the decode phase generates tokens one by one, is memory-access-intensive, and is latency-sensitive. The two have different GPU resource demand structures, and co-locating them on the same instance makes it difficult to optimize resource allocation for both simultaneously. PD separation splits the two workloads onto different instances or card groups, tuning batch and parallelism strategies separately, thereby improving overall throughput and reducing tail latency. DeepInfer currently does not provide this capability, which means it is difficult for it to finely allocate resources for the mixed characteristics of online traffic—one of the important reasons it is positioned as an offline engine.

Future Evolution and Industry Impact

DeepSig promises to continuously update the inference code for new models in this repository. If fulfilled, DeepInfer could evolve into an inference reference library that is updated in sync with model iterations.

From an ecosystem perspective, a high-performance inference engine open-sourced by a model vendor and directly derived from production will put pressure on the existing inference framework landscape: it both validates the scalability of vLLM as a foundation and demonstrates the upper limit of deep customization on top of vLLM. For teams using Ascend NPUs, DeepInfer's support for the 950 NPU is also a signal worth tracking.

Whether it is worth trying depends on your goal: if you want to study the engineering details of production-grade inference, its reference value is high; if you want to immediately obtain an inference service capable of handling online traffic, it currently lacks key components and requires self-built additions.

One-line Summary

DeepSig open-sources DeepInfer under Apache 2.0, revealing the engineering details and offline limitations of a production engine handling 155 trillion tokens per day.

Paragraph Summary

DeepSig has open-sourced DeepInfer, the high-performance inference engine powering its production services, under the Apache 2.0 license. The code is based on vLLM with custom kernels such as Flash MLA, Deep GMM, and Deep EP, and integrates mechanisms including speculative decoding, SWA bounded replay, and asynchronous zero-overhead scheduling. Production data shows a single cluster processing 155 trillion tokens per day with a 96% cache hit rate, indicating workloads heavily skewed toward high context reuse. The article notes that the core value of open sourcing lies in exposing the "invisible cost structure"—scheduling, caching, and kernel efficiency—for smaller teams to learn from. However, DeepInfer is currently an offline engine lacking a complete KV cache system, prefill/decode separation, and load balancing, so it cannot directly replace vLLM in production deployment and is better suited as a reference implementation for inference kernels and scheduling.