DeepEP Ascend Deep Dive: DeepSeek's MoE Communication Library Ported to Huawei Ascend NPU

Why MoE Needs a Dedicated Communication Library

Imagine an AI model housing 256 "experts," each specializing in a narrow domain. When a token arrives (a small chunk of text in the model's view, roughly half a word), a router glances at it and selects the 6 best-matched experts to handle it — the remaining 250 stay idle. This is the MoE (Mixture of Experts) architecture that has taken the large model world by storm.

Compared to traditional "generalist" models that activate all parameters, MoE works like a team of specialists: only the selected few experts are activated, making it faster and more efficient. But efficiency comes at a cost — communication gets complicated. A single card can't hold all 256 experts. Say you spread them across 8 cards with 32 experts each; the 6 selected experts will almost certainly be scattered across multiple cards. Every MoE layer then requires two cross-card data transfers — and DeepEP was built specifically to solve this problem.

DeepSeek^Ad has now ported this communication library to Huawei's Ascend NPU (an AI-specific accelerator chip that plays the same role as a GPU), releasing it open-source on GitHub under the name DeepEP Ascend. This article breaks down its core mechanisms based on a teardown video by a Bilibili creator.

All-to-All communication is one of the most expensive communication patterns in distributed training and inference. Unlike AllReduce (where all nodes aggregate the same data and broadcast it back), All-to-All requires each card to send different data blocks to every other card. Communication volume grows quadratically with the number of cards, and it's difficult to optimize with simple tree or ring topologies. MoE architectures naturally generate All-to-All demands: routing decisions are made independently on each card, but the selected experts are scattered across cards, requiring tokens to be precisely delivered to the right card, computed, then retrieved — with no shortcuts. This is precisely why NCCL's generic ncclAlltoAll often falls short in MoE scenarios — it has no understanding of "expert alignment," "FP8 compression," or "defer epilogue" semantics. DeepEP delivers deep customization at exactly this semantic layer.

Two Transfers: Dispatch and Combine

Cross-card communication in a MoE layer consists of two symmetric operations. The first is Dispatch: your token lives on card 0, but the matched expert lives on card 135 — Dispatch ships the data there. There's an efficiency detail: if two of the 6 selected experts reside on the same card, that card receives only one copy of the data, just like a courier making a single trip to one address with one package.

The second is Combine: after each expert finishes computing on its own card, it sends the result back along the same path to card 0, where the outputs are weighted by router scores (the influence weight the router assigns each expert) and summed to produce the token's final output. Together, these two transfers constitute one All-to-All communication round — like every student in a class simultaneously exchanging notes with everyone else.

Let's do the math: 16,384 tokens per card, each token is a vector of 7,168 values, 2 bytes per value, replicated 6 times — a single Dispatch pass moves 1.4 GB per card. Fortunately, FP8 (a lower-precision but more compact data format) cuts this in half.

API Compatible with NVIDIA, Entire Engine Replaced

DeepEP Ascend wasn't written from scratch. Its public API is fully aligned with the NVIDIA version of DeepEP — EP Buffer, Dispatch, Combine, even parameters like defer epilogue are identical. Code written for NVIDIA GPUs can be moved to Ascend with virtually no changes at the Python layer — but everything underneath is replaced:

• NVIDIA version: uses NCCL, NVLink for intra-node, RDMA for inter-node, kernels written in CUDA;
• Ascend version: uses HCCL (Huawei's communication library), UB for intra-node, URMA engine via HCCL/HCCN for inter-node, kernels written in Ascend C.

Same recipe, kitchen swapped from gas to induction — the cooking technique has to be relearned from scratch. Three specific differences: first, NUM_SMS refers to GPU SM count on NVIDIA but to AI Core count on Ascend (the actual compute units on the chip); second, Ascend requires the Dispatch Expand layout; third, the Ascend version currently operates in a single communication domain where all cards can communicate directly.

Another interesting design choice: on both platforms, device kernels aren't pre-compiled — they're compiled on first use (JIT). DeepEP only packages the host side (the CPU-side orchestration layer); kernels running on the accelerator are compiled on first execution and cached for reuse. Currently, the EP cross-card dispatch mechanism is the fully functional component, while PP, bucket, ngram, and other parallelism modes have API stubs reserved but implementations still in progress.

RDMA (Remote Direct Memory Access) is a key technology for high-speed cross-node communication. It allows one card's NIC to directly read and write data in another server's memory or device memory, completely bypassing the target server's CPU, with latency as low as microseconds. The NVIDIA ecosystem typically uses InfiniBand networks with RDMA; the Ascend equivalent is the URMA (Unified RDMA Abstraction) engine, which Huawei encapsulates within the HCCL/HCCN stack. URMA shares the same design philosophy as RDMA — SGE (Scatter-Gather Entry) describes where data lives, SQE (Send Queue Entry) issues the transfer instruction, and the NIC hardware handles the actual data movement autonomously. The CPU only issues the command and plays no part in moving data. This mechanism enables the "starting gun" phase of Dispatch to deliver data cross-card without CPU involvement.

Dispatch's Three Phases and FP8 Compression

Dispatch consists of three internal phases. Phase 1 — Queuing: incoming tokens check their routing manifest (topk_idx, listing which experts to visit), are "shadow-cloned" into 6 copies (one per expert) in the Dispatch Expand layout, and reordered by expert assignment. Each expert's queue is also padded to a multiple of 128 — called expert alignment — because the downstream grouped GEMM (matrix multiplication) only accepts 128-aligned inputs; empty slots are filled with zeros.

Phase 2 — Starting Gun: the kernel counts the tokens per expert and per destination card, fills out shipping manifests (SGE describes what's where and where it's going; SQE issues the instruction), and the URMA engine delivers the data directly to pre-allocated buffers on the destination card — entirely bypassing the CPU, which only fires the starting signal.

Phase 3 — Cleanup: once all data arrives, the epilogue kernel arranges it into its final form and generates a "treasure map" (receive source metadata) recording which slot holds data from which token and which routing entry — this map is what Combine uses to route results back. When the routing hasn't changed, a cached handle can be reused, skipping the full re-counting step.

Regarding FP8 compression: token data is compressed from BF16 to FP8 E4M3, halving the size. A per-token "instruction sheet" (scale factor) travels alongside each token and is used to reconstruct the original values at the destination. Test results comparing FP8 output against reference values confirm this phase passes numerical validation — but the README doesn't document end-to-end model accuracy impact, so no overclaiming there.

FP8 E4M3 is an 8-bit floating-point format using 4 bits for the exponent and 3 bits for the mantissa. Its representable range is far narrower than BF16 (16-bit), so naive truncation introduces significant error. In practice, it is used alongside per-token or per-tensor scale factors: find the maximum absolute value in the batch, compute a scaling ratio, multiply the original values by this ratio, then convert to FP8; the receiver uses the scale factor to reverse the process. DeepEP attaches one scale factor per token (as described — the "instruction sheet"), which is a per-token quantization granularity that preserves more precision than a single scale shared across an entire batch. This is also why FP8 is not lossless compression — the scale factors themselves occupy additional space, and the impact of quantization error on final model accuracy requires end-to-end validation. The README currently only covers numerical alignment for the Dispatch phase in isolation.

Computation-Communication Overlap: Board First, Ticket Later

The cleanup phase has a subtle trick embedded in it. By default, Dispatch completes synchronously — the CPU waits for the remote side to confirm all data has arrived. With defer epilogue enabled, the behavior changes to "board first, validate ticket later": Dispatch handles only the transport, deferring the epilogue cleanup until the data is actually needed — specifically, when event.current_stream.wait is called.

What happens in the gap? The README's canonical pattern is: fire off Dispatch, then compute the shared expert (a common "cafeteria" that every token visits regardless of routing, unrelated to inter-card communication). This computation neatly fills the waiting window, and by the time it's done you call wait to finalize the ticket. Communication and computation overlap seamlessly. In short: only the ticket-validation is deferred — the train itself is never late. Note: don't touch the data after current_stream.wait returns but before the buffer is released.

Combine: More Output, Same Cost

Combine is the reverse of Dispatch, with one extra step: computing the final score. After experts finish their computations, they follow the treasure map to ship 6 results back to their origin card, which weights and sums them using the top-k router weights. The epilogue can also piggyback a bonus operation: the shared expert's output can be added directly as a bias (a fixed ingredient folded in at the end) during the epilogue pass — done in one trip, saving a separate addition pass.

Benchmark Numbers and Caveats

Back to the opening question: if Dispatch and Combine are symmetric operations, why does Dispatch hit 373–375 GB/s on 8 cards while Combine only reaches 345–347 GB/s? The README explains — Combine carries the additional overhead of computing the weighted sum, and the URMA engine contends with HBM (the card's high-speed memory) for bandwidth. As the card count grows, this gap widens. At EP128, Dispatch drops to 313–320 GB/s and Combine to 272–278 GB/s; optimization of the Combine path is ongoing.

Test environment: Ascend 950DT, CANN 9.2.0, 16,384 tokens per card, hidden size 7168, 256 experts top-6, 32 AI Cores, FP8 for Dispatch and BF16 for Combine. As EP scales from 8 to 128, bandwidth decreases; at EP ≤ 32, Dispatch achieves 90–95% of the physical payload bandwidth ceiling.

Three important caveats:

1. This is POC-stage data. Measured using a proof-of-concept HDK (Huawei's hardware/software developer kit), not a production release. The README itself recommends waiting for the Q3 commercial HDK before using in production.
2. Timing is incomplete. Only enqueue and drain are measured; the final epilogue is excluded. Don't use these numbers to extrapolate end-to-end performance.
3. Combine is still being optimized, and Huawei firmware improvements are "in progress" — meaning on the roadmap, not yet delivered.

EP (Expert Parallelism) is a parallelism dimension unique to MoE models, alongside the more familiar data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP). In EP, different experts are assigned to different cards or nodes; each card holds the weights of only a subset of experts, and tokens are routed to the appropriate card based on routing decisions. The EP scale (EP8, EP32, EP128) is the number of cards participating in expert parallelism. A larger EP scale means each card holds more experts and uses less per-card memory, but also increases All-to-All hop count and latency — which is exactly why bandwidth decreases as EP scales up in this benchmark. In practice, EP scale must balance memory capacity, communication bandwidth, and compute efficiency. The data point that Dispatch sustains 90–95% of the physical bandwidth ceiling at EP ≤ 32 is a useful engineering reference.

Summary

DeepEP Ascend's value proposition in one sentence: it faithfully ports DeepSeek's "dispatch-and-combine" MoE communication mechanism to Ascend NPUs — same API, entirely new engine — cracking the communication bottleneck for large model inference on domestic AI hardware. It is currently at POC stage; the bandwidth figures are informative rather than binding commitments. A re-test after the commercial HDK ships would be worthwhile. But for the domestic AI chip ecosystem, this approach — maintaining API compatibility with the NVIDIA ecosystem while building a fully in-house implementation underneath — carries significance that shouldn't be underestimated.