[KongchangAI]
· 2 min read· 1,051 words

Fuse-All MoE: A Load-Balanced Inference Kernel on TPU

Fuse-All MoE: A Load-Balanced Inference Kernel on TPU

Fuse-All MoE combines kernel fusion and load balancing to boost MoE inference throughput on TPU by 34.5%.

This article introduces the Fuse-All MoE kernel from a Google research team, which jointly addresses two bottlenecks of deploying MoE on TPU: repeated HBM write-back of intermediate data and uneven expert load. Its core is fusing token gathering, expert projection, gated activation, and output write-back into a single kernel, keeping intermediates on-chip. Fixed token-slot destinations preserve routing semantics while allowing dynamic scheduling and expert placement. In PD-disaggregated architecture, the prefill phase uses EP16 scheduling and expert redistribution, reducing kernel runtime from 2.3x to 1.09x of the balanced reference under imbalanced load; the decode phase uses symmetric EP8TP2, cutting kernel latency by up to 21.3%. With serving stack optimizations, system throughput improves up to 34.5% over the Megablox EP16 baseline.

MoE (Mixture-of-Experts) architectures activate only a subset of experts for each token, allowing large language models to scale parameter capacity without proportionally increasing computation. But on TPU, this advantage is largely offset by two bottlenecks: intermediate data movement and uneven expert load. The Fuse-All MoE kernel proposed by a Google research team attempts to solve both problems simultaneously through a joint design of kernel fusion and load balancing.

Fuse-All MoE inference kernel diagram

Two Bottlenecks: Data Movement and Expert Load

MoE inference suffers efficiency losses from two sources. The first is intermediate data movement: token gathering, expert projection, gated activation, and output write-back are typically split into multiple independent kernel steps, with each step's results written back to HBM and read out again, resulting in poor on-chip cache utilization. The second is uneven expert load: the routing mechanism causes some experts to receive far more tokens than average, creating "hot experts" while other experts sit idle, wasting compute.

The core idea of Fuse-All MoE is to fuse token gathering, expert projection, gated activation, and output write-back into a single kernel, keeping intermediate expert computation results on-chip and avoiding repeated HBM round trips. This fusion strategy directly reduces kernel launch overhead and intermediate data spilling.

HBM and on-chip cache: HBM (High Bandwidth Memory) is the main memory of accelerators such as TPUs and GPUs. It offers large capacity but significantly higher access latency and energy consumption than on-chip SRAM. In MoE inference, if token gathering, expert projection, gated activation, and output write-back are each implemented as separate kernels, every step's intermediate results must be written back to HBM and read out again by the next step, creating multiple HBM round trips. On-chip caches (such as TPU vector memory/scalar memory and the SRAM near TensorCores) offer high bandwidth and low latency but limited capacity. The value of kernel fusion lies in keeping intermediate results on-chip and exchanging data with HBM only when necessary.

Hot experts and uneven load: The MoE router selects top-k experts for each token, but routing distributions are naturally imbalanced during both training and inference—a few experts may receive several times the average number of tokens. Under tensor parallelism or expert parallelism, a single hot expert becomes the long tail of the entire pipeline, while other experts and their corresponding chips sit idle waiting. The cost of uneven load is not insufficient total compute but rather that the slowest expert determines overall latency. Therefore, balancing strategies must redistribute computation without changing routing semantics.

Fixed Token Slots: Flexible Scheduling Without Changing Semantics

A design challenge introduced by fusion is: if expert placement and scheduling strategies change, how are routing semantics preserved? Fuse-All MoE's approach is to assign fixed token-slot destinations to tokens, so that the scheduler and expert placement can be adjusted dynamically while the token-to-expert routing mapping remains unchanged.

This design leaves room for subsequent load balancing—the system can rearrange expert computation tasks and distribute the load of hot experts without altering routing results.

Two Balancing Strategies in PD-Disaggregated Scenarios

In a prefill-decode (PD) disaggregated serving architecture, Fuse-All MoE adopts different balancing schemes for different phases.

The prefill phase uses an EP16 configuration, balancing workload through scheduling and expert redistribution. Experiments show that on a measured imbalanced EP16 prefill workload, kernel runtime dropped from 2.3x the balanced reference time to 1.09x—meaning most of the extra overhead caused by load imbalance was eliminated.

The decode phase uses a symmetric EP8TP2 configuration: the two TensorCore devices within each chip share active expert computation. Across the range of 32 to 512 decode tokens, symmetric EP8TP2 reduced kernel latency by up to 21.3% compared to the fused EP16 scheme.

PD disaggregation and EP/TP configurations: PD disaggregation means deploying the prefill phase (processing the entire input prompt at once) and the decode phase (token-by-token autoregressive generation) of large model inference in separate resource pools. The two phases have very different computational characteristics: prefill is compute-intensive large matrix computation, while decode is memory-intensive, latency-sensitive small-batch computation. EP (Expert Parallelism) distributes different experts across different chips; TP (Tensor Parallelism) splits a single matrix operation across multiple chips for cooperative execution. EP16 means experts are distributed across 16 devices, while EP8TP2 means 8-way expert parallelism combined with 2-way tensor parallelism, for a total of 16 devices. Symmetric EP8TP2 lets the two TensorCores within each chip share the same set of active expert computation, thereby improving per-chip utilization and load balance in small-batch scenarios like decode.

System-Level Gains: 34.5% Throughput Improvement

Kernel-level optimization also requires coordination with the serving stack. The paper notes that with accompanying serving stack optimizations, the overall system achieves up to 34.5% higher serving throughput compared to the Megablox EP16 baseline.

This figure shows that MoE inference optimization cannot remain at the level of individual operators. Kernel fusion, load balancing, scheduling strategies, and expert placement must be co-designed at the system level to truly harness TPU compute.

Megablox baseline: Megablox is a previously open-sourced MoE implementation from Google that provides a baseline set of kernels for expert routing and grouped matrix multiplication on TPU, and is commonly used as a comparison baseline for MoE inference optimization. It typically splits token gathering, expert computation, and result merging into multiple steps and does not perform deep scheduling optimization for uneven expert load, so it tends to expose HBM round-trip and long-tail latency issues in hot-expert scenarios. Fuse-All MoE reports a 34.5% throughput improvement over the Megablox EP16 baseline, indicating that the gains come not only from single-kernel fusion but also from scheduling and balancing coordination at the serving stack level.

Implications for MoE Inference Engineering

The design of Fuse-All MoE reveals a trend: the competitive focus of MoE inference is shifting from "expert computation itself" to "coordination among experts." With fixed routing semantics and flexible computation placement, this decoupling approach makes load balancing no longer an after-the-fact compensation but something that can be built into kernel scheduling.

For teams deploying MoE on specialized accelerators, this work offers two directions worth learning from: first, use kernel fusion to eliminate HBM round trips for intermediate data; second, treat load balancing as a scheduling problem rather than a hardware problem. The experience on TPU may not transfer directly to other hardware, but the combined approach of "fusion + balancing + decoupling" has broad reference value.

#Fuse-All MoE#MoE inference#TPU kernel fusion#load balancing#PD disaggregation#expert parallelism
Share:

Related articles