DeepSeek Open-Sources DeepGEMM for Ascend NPU: Domestic Chip Utilization Hits 99.8%

DeepSeek has made another move — this time targeting not a model or an inference framework, but the most fundamental layer of AI computation: matrix multiplication operators. The newly open-sourced project is called DeepGEMM for Ascend, a BLAS-level GEMM operator library designed to run on Huawei Ascend hardware. It reportedly pushes the Huawei 950DT chip's performance to 99.8% of its physical limit. For domestic AI compute, which has long been hampered by ecosystem gaps, this project carries far greater significance than an ordinary utility library.

What Is DeepGEMM, Exactly?

When people hear the name DeepGEMM, many assume it's yet another inference framework. It isn't. It's a BLAS-level low-level operator library that ignores inference framework concerns entirely and focuses on doing one thing: running matrix multiplication on AI accelerators as close to the physical limit as possible.

Its consumers aren't end users — they're open-source frameworks like vLLM and SGLang that handle model inference and training. These frameworks rely internally on high-performance matrix multiplication operators for their linear layers and MoE expert layers. That's exactly what DeepGEMM serves.

It's worth emphasizing that DeepGEMM is not an entirely new project. DeepSeek^Ad had previously open-sourced a DeepGEMM optimized for NVIDIA GPUs. This Ascend edition keeps the API interface identical — only the compute backend has been ported to Huawei's AI chip. In other words, existing call patterns require virtually no modification.

Background: BLAS & GEMM BLAS (Basic Linear Algebra Subprograms) is a specification defining standard interfaces for vector and matrix operations, originally developed in 1979 for scientific computing. GEMM (General Matrix Multiply) is the most fundamental operation within it — virtually every linear transformation in neural networks, including fully connected layers, QKV projections in attention mechanisms, and output projections, is essentially a GEMM. The reason modern AI frameworks run efficiently across different hardware largely comes down to highly optimized BLAS libraries from each vendor: NVIDIA has cuBLAS, Intel has MKL, and Huawei Ascend has historically had comparatively weak open-source optimization at this layer. That's precisely why a dedicated library that pushes GEMM performance close to hardware peak is critically important for building a complete inference software stack.

Why MoE Models Need It Especially

To appreciate the value of this operator library, you first need to understand where the inference bottleneck lies in models like DeepSeek.

Breaking it down, the bottleneck concentrates in two places. The first is the read/write of KV Cache in the attention mechanism — especially during the decoding phase, which is highly susceptible to memory bandwidth limits. The second is the GEMM computation in MoE routed experts, which is compute-intensive but irregularly shaped, since tokens are dynamically assigned to different experts and matrix dimensions are not fixed. This is precisely what makes optimization difficult.

DeepGEMM for Ascend offers quite comprehensive operator coverage, supporting multiple precision combinations including FP4, FP8, and BF16 — such as FP4×FP4, FP8×FP4, FP8×FP8, and BF16×BF16 — serving all standard linear layers.

Beyond general dense matrix multiplication, the project includes several operators specific to DeepSeek models: Mgroup GEMM handles matrix multiplication for MoE expert assignments; MQA Logits is a sparse attention scoring operator used in DeepSeek's Lightning Indexer (DSA); and MGroup MoE serves large-scale MoE model deployments. These last three operators are deeply tied to the specific architectures of new models like DeepSeek V4.1 Flash.

Background: MoE Architecture MoE (Mixture of Experts) is a sparsely activated neural network architecture: instead of activating all parameters for every token, a router dynamically selects a small number of "expert" sub-networks to process each token. This means that within the same batch, different tokens are routed to different experts — making the actual matrix shapes dynamic. Some experts receive many tokens, others very few or none at all. This "irregular" batching pattern significantly reduces the efficiency of traditional GEMM libraries optimized for fixed large matrices, requiring specially designed operators capable of handling variable shapes (grouped GEMM). Models like DeepSeek V3 and R1 make heavy use of MoE architecture, which is the fundamental reason the Mgroup GEMM operator in DeepGEMM is so important.

How Does It Achieve 99.8% Utilization?

The near-physical-limit utilization isn't an empty claim — the project documentation reveals the engineering details behind it.

At its core, it calls Ascend's low-level matrix interfaces directly, with extensive optimizations around fractal matrix data layouts, address alignment, and dimension constraints. These low-level operations normally require a large amount of manual offset calculations and verbose low-level parameters and kernel signatures. DeepGEMM builds a thin abstraction layer on top of all this.

The thickness of this abstraction is a critical tradeoff: too heavy and it becomes a performance killer; too thin and it's hostile to continuous integration. DeepGEMM keeps the kernel code clean while minimizing performance overhead.

The project also features two design choices that take advantage of specific Ascend hardware capabilities. First, it leverages the hardware's asynchronous data loading to skip unnecessary data transfers. Second, it uses a skewed pipeline that organizes the overlap of computation and data movement to approach hardware limits.

Benchmark results are impressive: BF16×BF16 achieves 99.8% hardware utilization, while FP4×FP4 reaches 98.3%. The Huawei 950DT itself is no slouch in specs either — it delivers roughly 1.7 PFLOPS in FP4 compute.

Filling a Critical Gap in China's AI Ecosystem

Huawei Ascend hardware has never been particularly weak. Where it has genuinely lagged NVIDIA is in software ecosystem. NVIDIA built a massive open-source ecosystem on top of CUDA, while Huawei's own CANN ecosystem, despite its breadth, has had gaps when it comes to supporting mainstream large models like DeepSeek.

This open-source release precisely addresses the missing low-level operator layer for large models. Looking at the technology stack, DeepGEMM for Ascend is built on two underlying projects: TileLang, a domestic open-source operator optimization project, and DeepSeek's own DJIT, combined with Huawei's C kernel low-level code, all wrapped into this open-source library.

There's also a telling signal here: there had been rumors that DeepSeek had purchased a large number of Huawei's latest AI chips, and this release somewhat confirms it — they've clearly done extensive testing on the 950DT to be able to release a compatible, high-performance set of kernels.

The project has just launched, and frameworks like vLLM haven't yet integrated DeepGEMM into their Ascend optimization paths. Once that integration happens, inference performance for DeepSeek-class models on the Huawei Ascend 950DT is expected to see a significant boost.

Background: CANN Ecosystem CANN (Compute Architecture for Neural Networks) is the software stack Huawei built for Ascend NPUs, analogous to NVIDIA's CUDA ecosystem, covering drivers, operator libraries, compilers, and framework adaptation layers. Compared to CUDA, CANN's core challenges have been: weak open-source community accumulation, few third-party operator optimization libraries, and lagging maintenance of Ascend backends in mainstream inference frameworks. TileLang is a tile-abstraction operator programming language developed by a Chinese team to lower the barrier for writing high-performance kernels; DJIT is DeepSeek's internal JIT compilation infrastructure for generating and optimizing hardware-specific instructions at runtime. DeepGEMM combines both with Huawei's low-level C kernels, effectively building a high-performance operator bridge on top of CANN specifically for large-model inference — enabling Ascend hardware to better serve mainstream open-source models like DeepSeek.

Conclusion

For China's domestic compute ecosystem, catching up in hardware is relatively straightforward; the real difficulty lies in the ecosystem. DeepGEMM for Ascend addresses the low-level operator gap for large models on Huawei Ascend NPUs — with near-zero migration cost and near-physical-limit performance. As upstream frameworks gradually integrate it, the cost-performance profile of the Ascend platform is expected to improve further. That may ultimately be the most far-reaching value of this foundational operator library.