AMD MoRI + vLLM: Wide Expert Parallelism and RDMA KV Cache Transfer for MoE

AMD Unveils a New MoE Serving Approach at PyTorch Conference

Three AMD engineers — AI model researcher Ravi Gupta, GPU software developer Rishi Madhuri, and AI engineer Shiksha — will present a poster at PyTorch Conference 2026 focused on Wide Expert Parallelism and RDMA KV cache transfer for efficient disaggregated MoE model serving on AMD hardware.

Based on publicly available preview information, the core of this work is integrating MoRI (AMD's relevant technology stack) with the widely used inference framework vLLM, offering an optimized path to address the performance bottlenecks that Mixture-of-Experts (MoE) models face at large-scale deployment.

Why Wide Expert Parallelism Matters

The MoE architecture has become the dominant approach for scaling large model parameters in recent years. By splitting a model into multiple "expert" sub-networks and activating only a subset during inference, MoE dramatically increases model capacity while keeping computational costs in check. However, this architecture introduces new engineering challenges in production: experts are distributed across different devices, and how efficiently tokens are routed between nodes and data is synchronized directly determines serving throughput and latency.

Wide Expert Parallelism is a strategy designed to address exactly this problem — it spreads experts across more devices horizontally to reduce per-device load and increase overall parallelism. AMD's integration of this approach with vLLM means the solution has the potential to be more broadly adopted and validated within the open-source ecosystem.

It's worth noting that MoE architectures face a unique challenge known as Expert Load Imbalance: the activation frequency of different experts varies dramatically, easily leading to some devices sitting idle while others are overloaded. Wide Expert Parallelism scales expert distribution horizontally across devices. Unlike traditional tensor parallelism or pipeline parallelism, it focuses on having more devices each host independent expert weights, enabling load balancing across a larger device pool during token routing. This strategy is especially critical for models with a very large number of experts (e.g., DeepSeek^Ad-V3's 256 experts) — since only a few experts are activated per inference pass, wider parallelism means popular experts can be replicated across multiple cards to distribute pressure, while less-used experts don't consume additional memory. The trade-off is that inter-node communication grows linearly with parallelism width, making high-bandwidth interconnects essential.

The Significance of RDMA KV Cache Transfer

In a disaggregated serving architecture, the prefill and decode phases are typically split across different hardware resources, requiring efficient transfer of KV caches between stages. KV caches are often large in volume, making transfer efficiency a critical factor in end-to-end latency.

RDMA (Remote Direct Memory Access) technology allows devices to access remote memory directly, bypassing the CPU, which significantly reduces data transfer latency and overhead. AMD's use of RDMA for KV cache transfer is aimed at resolving the cross-node data movement bottleneck in disaggregated MoE serving — this is especially important for production-grade inference services that demand low latency and high concurrency.

Disaggregated serving has been a major trend in high-performance inference architecture over the past two years. Its core idea is to decouple the prefill phase (compute-intensive) from the decode phase (memory bandwidth-intensive) — two stages with fundamentally different computational characteristics — and assign them to separate hardware resource pools that can scale independently. This avoids the mutual interference and poor resource utilization that occurs when both tasks share the same hardware. However, decoupling introduces new data movement requirements: KV caches generated by prefill nodes must be transferred to decode nodes in real time, and for long-context or large-batch requests, a single transfer can amount to several gigabytes of KV cache. RDMA bypasses the OS kernel and lets the NIC directly read and write remote GPU memory, compressing transfer latency from the microsecond-level overhead of the TCP stack down to the nanosecond range, while significantly reducing CPU utilization. AMD's ROCm ecosystem enables this capability through support for RDMA over Converged Ethernet (RoCE), competing with InfiniBand solutions and giving AMD-based clusters a competitive foundation for data transfer in disaggregated inference scenarios.

AMD's Broader Ecosystem Play

This work is being advanced by researchers and engineers within AMD, reflecting the company's sustained investment in the AI inference software stack. For a long time, discussions around large model inference optimization have been concentrated within specific vendor hardware ecosystems. Through deep integration with open-source frameworks like vLLM, AMD is attempting to broaden its influence in the AI serving space.

For developers focused on MoE deployment and heterogeneous hardware inference, this poster presentation is worth watching. It brings together three core topics in MoE serving — expert parallelism, disaggregated architecture, and KV cache transfer — into a single practical solution targeting the AMD platform.

Summary

The publicly available information is still largely preview-level; specific performance numbers, architectural details, and implementation code are expected to be disclosed at the PyTorch Conference 2026 event itself. For teams looking to deploy large-scale MoE models on AMD hardware, this solution combining MoRI and vLLM offers a compelling technical direction worth following.