Great News for Older GPUs! DeepSeek On-Device Inference Throughput Doubles

The compute barrier for large model inference is being chipped away by community efforts. Recently, an optimization for DeepSeek on-device inference has drawn attention on Bilibili: throughput on ordinary GPUs has more than doubled, long-context scenarios have gone from "frozen" to running at full speed offline, with almost no loss in accuracy, and data stays entirely on the local device. This opens up new possibilities for running large models on consumer-grade GPUs.

Two Key Technologies: Dynamic VRAM Paging and Mixed-Precision Operators

According to the Bilibili creator, the core of this breakthrough lies in the introduction of dynamic VRAM paging and a new generation of mixed-precision operators. The former manages VRAM pages more flexibly, reducing waits and waste caused by VRAM fragmentation; the latter performs computations in mixed precision while maintaining numerical stability, lowering compute overhead. Combined, they "completely squeeze out" the fragmentation overhead that was once such a headache.

From a technical perspective, dynamic VRAM paging resembles the idea of virtual memory in operating systems, freeing VRAM allocation from the need for large contiguous blocks and thereby improving utilization. Mixed-precision operators apply different numerical precisions to different computation stages—retaining high precision where it matters and lowering precision where it's not sensitive in exchange for speed. This is not an entirely new concept, but adapting it to DeepSeek^Ad's model structure for on-device tuning is where the value of community optimization lies.

Dynamic VRAM paging borrows the paging mechanism of OS virtual memory. Traditional inference frameworks typically require large contiguous blocks when allocating VRAM; in long-context or dynamic batching scenarios, external fragmentation easily occurs—even if total free VRAM is sufficient, the next allocation request cannot be satisfied, leading to OOM or forced waiting. The paging approach divides VRAM into fixed-size pages and maps logical addresses to physical pages via a page table, so allocation no longer depends on contiguity. Combined with on-demand swapping (similar to swap), pages not in use can be offloaded to host memory when VRAM is tight, supporting longer contexts within limited VRAM. The cost is the extra overhead of page-table lookups and swapping, which requires fine-grained scheduling policies to balance.

Mixed-precision operators refer to mixing different precisions such as FP16/BF16 and FP32 within the same forward computation: compute-intensive operators like matrix multiplication and attention run in low precision to boost throughput and reduce VRAM bandwidth pressure, while numerically sensitive stages like LayerNorm, Softmax, residual accumulation, and optimizer states retain FP32 to avoid overflow and accumulated rounding errors. BF16, with the same exponent bits as FP32 and a large dynamic range, is often more robust than FP16 in on-device large models. Implementation typically relies on tensor cores and automatic mixed precision (AMP) frameworks, along with techniques like loss scaling. The gains depend on whether the GPU supports the corresponding low-precision instructions; older cards may not achieve the same acceleration.

Measured Performance: Throughput Doubles, Long Context No Longer Freezes

According to tests, offline inference speed more than doubles on a single consumer-grade card. A notable detail is the long-context scenario: previously limited by VRAM and fragmentation overhead, long-context inference would easily freeze or run extremely slowly; now it can proceed offline at full speed. This means users can process longer documents, codebases, or multi-turn conversations locally without relying on cloud APIs.

On accuracy, the creator says there is "almost no accuracy loss." This is crucial—the practicality of on-device inference depends not only on speed but also on whether output quality is acceptable. If optimization caused a noticeable drop in accuracy, it would not be worth it for tasks like code generation and logical reasoning. So far, the approach strikes a good balance between speed and accuracy.

Data Never Leaves the Device: The Dual Appeal of Privacy and Cost

Another highlight is that data never leaves the local device. For enterprises and individual users dealing with sensitive information, local inference means no need to upload data to the cloud, avoiding privacy leak risks. At the same time, offline operation saves API call costs, offering a clear cost advantage in the long run.

Of course, the price of local inference is maintaining your own hardware and software environment. Community optimizations lower the hardware barrier, but deployment and tuning still involve some technical threshold. However, as on-device adaptive operators continue to evolve, this barrier is expected to decrease further.

The Significance and Limitations of Community Optimization

This optimization comes from the community rather than the official team, reflecting the vitality of the open-source model ecosystem. Deep tuning by the community for specific models and specific hardware often uncovers optimization space not yet covered officially. Dynamic VRAM paging and mixed-precision operators are not unique to DeepSeek, but adapting them to its architecture requires deep understanding of the model's computation graph and VRAM access patterns.

It should be viewed rationally: the information currently comes from a single creator's share, with no broader third-party reproduction or official confirmation yet. The specific test conditions for the "doubled" throughput (GPU model, context length, batch size, etc.) are not disclosed in detail, and actual gains may vary by scenario. In addition, on-device inference still cannot match large-scale cloud clusters in absolute performance; it is better suited to scenarios that are privacy- and cost-sensitive with controllable workloads.

On-device adaptive operators refer to optimization methods that automatically select operator implementations, block sizes, precision strategies, and VRAM layouts for a specific model structure and target hardware at compile time or runtime. Unlike general-purpose operator libraries, they require understanding the model's computation graph (e.g., attention head dimensions, KV Cache access patterns) as well as the hardware's VRAM hierarchy, cache capacity, and instruction set characteristics. Common techniques include operator fusion, KV Cache paging management, and falling back on precision based on hardware capabilities. Such optimizations are usually strongly tied to specific GPU models and context lengths; gains may change after switching hardware or model versions, so test conditions must be clearly stated when reproducing.

How Far Are Local GPUs from Replacing the Cloud?

The progress of on-device inference is evident, but "fully replacing cloud services" is still unrealistic. Cloud clusters have inherent advantages in throughput, concurrency, and model scale, and can continuously upgrade hardware. Local consumer-grade GPUs are limited by VRAM capacity and compute power, making them better suited to small-to-medium models or specific tasks. However, for many individual developers and small teams, local inference can already cover a considerable portion of needs—especially now that optimization has more than doubled throughput.

In the future, as on-device adaptive operators continue to evolve and more community optimizations emerge, the gap between local and cloud will narrow further. But the two are more likely to be complementary: the cloud handles heavy-load, high-concurrency tasks, while local handles privacy-sensitive, low-latency, offline scenarios. For users with "older GPUs," this wave of optimization is undoubtedly good news—no need to rush to upgrade hardware, and existing devices can deliver more value.

One-line Summary

Community optimization doubles DeepSeek throughput on ordinary GPUs, with long context no longer freezing.

Paragraph Summary

The DeepSeek on-device inference optimization shared by a Bilibili creator uses dynamic VRAM paging and a new generation of mixed-precision operators to more than double offline inference throughput on a single consumer-grade card, turning long-context scenarios from frozen to full speed with almost no accuracy loss, and keeping data entirely local. Dynamic VRAM paging borrows from virtual memory to reduce fragmentation waste, while mixed-precision operators retain high precision where it matters and lower precision elsewhere in exchange for speed. The approach highlights the value of community deep tuning for specific models and hardware, but the information comes from a single source, test conditions are not detailed, and actual gains vary by scenario. Local GPUs and the cloud are more likely to be complementary.