DeepSeek Open-Sources DeepInfer Inference Framework: How Long-Context Reuse and Speculative Decoding Accelerate AI Programming

DeepSeek has open-sourced an inference framework called DeepInfer, which racked up nearly 13K GitHub stars in less than a day. It targets some of the most resource-intensive parts of LLM inference: how to reuse long contexts, how to allocate compute across multiple requests, and how to reduce waiting time during response generation. These sound low-level, but they directly determine the waiting experience when you use AI to write code or modify pages.

According to a breakdown by a technical creator on Bilibili, DeepInfer is not a complete online service system but an open-source offline inference engine and related core capabilities. Full production deployment still requires components like caching systems and load balancing, and the relevant code has not yet been fully released.

What Happens on the Server During a Single Page-Edit Request

Suppose you ask AI to help modify a page and send it the project files along with a screenshot of the page. After the first version is done, you add just one sentence: "Move the login button to the top right." The new requirement is only one sentence, but the content sent to the model in the background often still carries the earlier code files, conversation history, and the results just produced in the previous round. This content was clearly already computed last round—does it really need to be recomputed from scratch this round?

Breaking down the processing flow of a single request makes the optimization points clear. The original image first goes through visual encoding to convert it into a representation the model can process, then the model processes the input context sent this round—this stage is called prefill. Only after processing the input does it enter the decode stage, generating subsequent content step by step.

The computational characteristics of these two stages are completely different: prefill processes a large chunk of input material in one go, while decode generates subsequent tokens one after another. If everything is crammed onto the same set of machines, resources must be coordinated back and forth. DeepSeek^Ad's described production solution separates visual encoding, prefill, and decode into separate deployments, each with its own machines—when the input is long, allocate more compute to the context-processing stage; when many requests are waiting for generation, allocate more resources to the generation stage. What users perceive as "how long until it starts answering" and "how fast the output is after it starts" need to be optimized separately in the backend.

Prefill and decode are the two stages of autoregressive LLM inference. In the prefill stage, the complete user input context (such as code files, conversation history, and image representations after visual encoding) is fed into the model at once, and the attention and KV Cache for all input tokens are computed in parallel. This is compute-intensive, and the time consumed mainly depends on input length. The decode stage starts from the already-processed context and autoregressively generates output token by token, processing only one new token per step. This is memory-bandwidth-intensive, and the time consumed depends on output length and hardware memory access efficiency. Separating the two deployments allows independent scaling for different resource bottlenecks and avoids contention for resources.

Cross-Machine KV Cache Reuse: Don't Let Last Round's Computation Go to Waste

After the first version of the page is done, you add that request to move the button. The earlier project files haven't changed, and the conversation history contains a large amount of identical content. If the results computed last round could be reused, some redundant computation could be avoided.

When the model processes context, it produces a type of cache called KV Cache, which stores intermediate results needed for subsequent computation. It caches computational state, not a ready-made full answer stored in advance, so when a new requirement comes in, the model still has to continue generating based on the current context. But as long as the identical prefix of the request meets the reuse conditions and the cache is still there, there is an opportunity to continue computing from the existing results.

The problem is that an online service is handling many requests at the same time. When you send the second-round task, you may not be assigned to the same machine as before. If the cache only remains on the original machine and that machine happens to be busy, while another idle machine cannot access the results just computed, you have no choice but to wait or recompute—this is exactly the classic challenge of cache reuse in inference services.

DeepSeek's production solution uses the 3FS storage system to place the main type of KV cache in independent shared storage, so other nodes handling context can also read existing caches and don't have to wait indefinitely for the original machine just to hit the cache.

Reading and transmission themselves also have costs, and some state is stored in other ways. How much time can actually be saved depends on how much content is hit this time and how long cache reading versus recomputation each takes. This is not an unconditional acceleration solution but a trade-off that must be dynamically weighed.

KV Cache is a key intermediate state in LLM inference. When generating each token, the Transformer needs to compute attention based on all tokens processed so far, and each layer stores key (Key) and value (Value) vectors for the processed tokens. When generating new tokens later, these vectors are reused directly to avoid recomputation. In long-context scenarios, the size of the KV Cache can reach tens of GB, and its storage location and hit efficiency directly determine inference latency and GPU memory usage. Therefore, whether KV Cache can be reused across requests and across machines has become one of the core problems in optimizing long-context inference services.

Speculative Decoding and DSpark: How the Generation Stage Can Advance Multiple Steps at Once

Once the reusable parts of the context have been reused and the new requirement has been processed, it's time for the model to generate the modified code or the next tool call. How else can the generation step be accelerated? DeepInfer uses speculative decoding: first let a draft model propose several candidate tokens, then have the main model verify them together. The verified prefix is kept directly; when an unacceptable position is encountered, it is corrected according to the verification rules, and the draft tokens following it cannot be copied directly either. In this way, the main model has a chance to advance multiple tokens in a single computation round.

But more drafts are not always better. If the later ones often fail verification, both generating and verifying them waste compute. DSpark in DeepInfer first estimates the probability that drafts will be accepted, then combines that with hardware throughput to adjust how long a draft is worth verifying for each request—advance more where prediction is easy, verify less where confidence is low.

Whether it ultimately becomes faster still depends on the load and acceptance conditions at the time. The benefits of speculative decoding are highly dependent on task type and model behavior. In scenarios with strong structure such as code generation, draft acceptance rates are usually higher and the benefits are more obvious.

Speculative Decoding is a technique for accelerating generation without changing the output distribution of the main model. The basic idea is: use a smaller, faster draft model to autoregressively generate several candidate tokens first, then feed these candidate tokens together with the original input into the main model for a single forward computation, where the main model verifies in parallel whether each candidate position is accepted. The longest verified prefix is adopted directly; the first unaccepted position is regenerated by the main model according to the sampling rules, and subsequent drafts are discarded. Because the main model can verify multiple tokens in one forward pass, generation throughput improves significantly, but the benefit depends on the draft model's acceptance rate—the higher the acceptance rate, the more obvious the acceleration.

Practical Significance for AI Programming and Agent Developers

Putting this page-modification task together: input processing and output generation can be allocated resources separately, previously computed context can be reused when the cache is hit, and when generating new content, drafts can first be proposed and then verified by the main model. These optimizations all happen during the process of the model being called again and again.

It should be made clear that this content is organized based on official documentation and did not include deployment speed tests. What DeepInfer currently open-sources is an offline inference engine and related core capabilities. If you are developing Agents, even if you cannot use this deployment for now, understanding these processes is very helpful: the longer the context and the more invocation rounds, the more worth paying attention to redundant computation and request scheduling. When analyzing why a call is slow, you can also start separately from the two stages of input processing and output generation.

From an industry perspective, DeepSeek open-sourcing the core capabilities of its inference engine means that optimization methods once hidden inside various inference services—long-context reuse, disaggregated deployment, and speculative decoding—are becoming public knowledge. For small and medium teams and independent developers, this lowers the barrier to building efficient inference services themselves; for the inference framework track, the focus of competition is shifting from "whether it can run" to "whether it runs economically and fast."