Breaking Down DeepEP and DeepGEMM: MoE Communication and Matrix Computation Acceleration in Practice

The DeepSeek^Ad community recently open-sourced two low-level acceleration projects targeting the Ascend platform — DeepEP Ascend and DeepGEMM Ascend. The former focuses on solving cross-device communication challenges in Mixture-of-Experts (MoE) models, while the latter targets matrix computation optimization in large model training and inference. Both libraries are the product of deep collaboration between Huawei Ascend and the DeepSeek team, pushing hardware to its theoretical limits along two key dimensions: compute throughput and memory bandwidth. This article breaks down their design philosophy and performance gains layer by layer.

Hardware Foundation: Atlas 950 SuperPod and UB Interconnect

DeepEP's performance depends heavily on the underlying interconnect capabilities. The entire solution is built on the Atlas 950 SuperPod, leveraging its high-speed interconnect for cross-device data dispatch, result aggregation, and collective communication operations in MoE models.

In terms of topology, a single PC16 node consists of two 950 CPUs paired with 16 NPUs, with every 8 cards forming a 4-Mesh chassis. Within a chassis, each NPU connects directly to the other 7 cards via 7 UB ports, achieving a bidirectional bandwidth of 784 GB/s. An additional 8 UB ports are dedicated to cross-chassis communication, providing 850 GB/s bidirectional bandwidth. Multiple PC16 nodes can be further combined into a UB2128 super-node.

The most significant architectural shift here is the interconnect paradigm: cards are connected at the bus level, rather than through the traditional "NIC + RDMA" path. This bus-level direct connection is what enables the extreme bandwidth utilization achieved later. In critical training communication workloads, communication bandwidth reaches over 90% of the theoretical maximum.

What is UB? UB (Unified Bus) is Ascend's proprietary high-speed chip-to-chip interconnect technology, analogous in role to NVIDIA's NVLink but implemented differently. Traditional GPU cluster communication relies on InfiniBand or RoCE networks, where data must traverse a NIC and the network protocol stack before reaching the destination — introducing protocol overhead and latency. UB directly connects multiple NPUs in a fixed topology, bypassing the network protocol stack entirely for lower latency and more stable bandwidth. The 4-Mesh topology means every card within a chassis has a direct link to each of the other 7 cards, eliminating the bandwidth penalty of multi-hop forwarding. This bus-level interconnect makes communication behavior precisely predictable and schedulable in software — which is the fundamental reason DeepEP can sustain over 90% bandwidth utilization approaching the theoretical limit. Traditional network solutions typically achieve only 60–70% due to network jitter and protocol overhead.

DeepEP's Four Buffer Types and EP Buffer Optimization

Building on this hardware foundation, DeepEP provides four types of Buffers, each with a distinct role:

• EP Buffer: Handles Expert Parallelism Dispatch and Combine operations, with support for the FP8 data type and Cache Handle operations;
• PP Buffer: Manages Send/Receive operations in pipeline parallelism;
• Ingram Buffer: Handles table writes and per-layer fetches under the Ingram algorithm structure;
• Bucket Buffer: Manages bucketing-related operations.

Among these, the EP Buffer is the centerpiece of the performance design. Its core objectives are extreme local processing performance and communication performance, leveraging UMA to advance communication in the background and free up AIV cores, then overlapping computation with communication to improve end-to-end performance.

Local Acceleration for Dispatch and Combine

Dispatch requires statistics gathering, deduplication, and packet assembly before communication begins. Combine can perform local reduction before sending data. Post-communication, there is also a cleanup data processing phase (the Applog stage).

For local processing, DeepEP does three things: parallel multi-core acceleration for statistics and deduplication, batch packet assembly to reduce data preparation time, and multi-buffered pipelining in the cleanup stage to improve write-back bandwidth utilization.

Payload and Concurrency Optimization on the Communication Side

The communication-side strategy centers on "increasing effective payload + multi-queue concurrency". Concretely, multiple token segments and their source data are packed into a single transfer to maximize payload size, while multiple communication queues concurrently exploit bandwidth across multiple interconnect ports. Combine's pre-communication local reduction also effectively reduces the amount of data that needs to be transmitted.

Combined with UMA advancing communication in the background and freeing AIV cores, communication and computation overlap effectively — ultimately achieving over 90% bandwidth utilization approaching the theoretical limit in critical communication workloads.

Background: Expert Parallelism and its Communication Primitives Expert Parallelism (EP) is a parallelism dimension unique to MoE models. In dense models, every token passes through the same FFN layer. In MoE models, each token is routed to only a few experts (FFN sub-networks), which are typically distributed across different devices. This creates two communication primitives: Dispatch — sending each token to the device hosting its assigned expert; and Combine — aggregating and weighted-summing the expert outputs back to the token's origin device. These two communication steps occur frequently at scale in MoE inference, and their communication volume grows linearly with the number of experts and tokens, making them a primary end-to-end performance bottleneck. The EP Buffer is designed specifically to optimize these two primitives to the extreme. UMA (Unified Memory Access) allows communication to be driven in the background by dedicated hardware units while simultaneously releasing AIV (AI Vector) compute cores to do other work, enabling true overlap of communication and computation.

DeepGEMM Ascend and the MegaMoE Fused Operator

DeepGEMM Ascend covers low-precision computation for large model training and inference, along with several specialized fused operators including grouped matrix multiplication, tensor computation, logits computation, RMSNorm, and the most representative one in this release: MegaMoE. Under deep optimization by the DeepSeek team, each fused operator nearly saturates the hardware's compute and bandwidth capacity.

The core design of MegaMoE is to place communication and computation inside the same operator, pipelined at the block level. It uses 1 AIC plus 2 AIV cores working in coordination:

• AIV0: Responsible for weight preparation — parsing and preprocessing source data, reading FP4 weights, converting them to FP8 NZ format within the UB, then loading into L1 for matrix computation;
• AIC: Responsible for the GMM computation in Linear1 and Linear2;
• AIV1: Responsible for Dispatch fetching and activation computation — after Linear1 completes, AIV1 executes SwiGLU, expert weighting, and FP8 quantization, then hands off to Linear2; after Linear2 completes, AIV1 handles the Combine writeback and writes activations to remote devices across cards.

The final Combine reduction (Sum computation) is completed jointly by AIV0 and AIV1. The essence of the entire scheduling strategy is: block-level Dispatch fetching drives downstream GMM computation, Linear1 and Linear2 are interleaved to maximize Cube utilization, multi-buffer and L1 multi-level prefetching improve MAC utilization, and a ring-shaped workspace reduces memory management overhead.

Performance Gains

With these optimizations, the MegaMoE operator achieves a hardware utilization level of 852 TFLOPS. Compared to the baseline (serial scheduling of Dispatch + MoE FFN + Combine):

• Training scenarios: over 1.5× improvement;
• Decode scenarios: over 2.5× improvement.

The larger gain in Decode is primarily due to eliminating per-operator scheduling overhead by consolidating everything into a single operator ("one slot"), as well as reducing kernel launch preamble costs.

Background: Ascend Core Types, FP8/FP4, and TFLOPS Ascend NPUs have three types of compute cores: AIC (AI Core, also called the Cube core) is specialized for high-throughput matrix multiplication (GEMM) and is suited for large, regular compute blocks; AIV (AI Vector Core) handles element-wise operations, activation functions, normalization, and other vector computations; AICPU handles scalar control logic. FP8 is an 8-bit floating-point format that reduces memory footprint and bandwidth consumption by half compared to FP16, while AIC throughput for FP8 can be twice that of FP16 — making it a critical data type for low-precision large model inference. FP4 compresses weight storage even further but requires conversion to FP8 format before computation (NZ format is Ascend's proprietary matrix storage layout, transposition-optimized to match the Cube core's data access patterns). AIV0's role in MegaMoE — completing the FP4→FP8 NZ conversion — is precisely designed to keep the weight decompression and matrix computation pipeline fully loaded, preventing the AIC from stalling while waiting for data.

TFLOPS (Tera Floating Point Operations Per Second) is the core metric for measuring AI accelerator utilization. The measured value of 852 TFLOPS needs to be understood in relation to the hardware's rated peak throughput — this figure indicates that the MegaMoE operator keeps the Ascend NPU's matrix compute units at an extremely high utilization level, essentially saturating available compute capacity. The Decode scenario shows a larger improvement than training (2.5× vs 1.5×) because Decode generates only one token at a time, so the fixed overhead of operator scheduling (launch latency, kernel load time) represents a much higher fraction of total execution time compared to training's batch processing. Merging Dispatch, MoE FFN, and Combine from three separate operators into one MegaMoE operator reduces three independent operator launch overheads to one — a benefit that is especially pronounced in Decode, where compute volume is relatively small and scheduling overhead is relatively prominent.

Conclusion: A Hardware-Software Co-design Ecosystem

The core value of DeepEP Ascend and DeepGEMM Ascend lies in tackling MoE's two major bottlenecks — cross-device communication and matrix computation — individually and pushing each to the hardware limit. The former achieves over 90% bandwidth utilization through bus-level interconnect and communication/computation overlap; the latter delivers 1.5–2.5× performance gains through fused operators and fine-grained scheduling.

These two projects also represent a landmark example of deep co-engineering between Huawei Ascend and the DeepSeek team to build out the AI chip software ecosystem. For developers focused on low-level large model engineering optimization, they offer a concrete reference implementation of the "hardware-software co-design to extract maximum hardware potential" paradigm.