PixelMoE
A pixel-space diffusion model that adopts a standard LLM-style sparse MoE scaling architecture, combining Token-Choice routing with pixel-space hierarchical flow matching to expand total model capacity at a fixed number of active parameters.
Timeline (last 90 days)
PixelMoE proves that the Mixture-of-Experts (MoE) architecture, mature in the LLM field, can be directly transferred to pixel-space diffusion models without any diffusion-specific routing mechanism.
PixelMoE directly adopts the standard Token-Choice routing from LLMs, keeps one shared expert, and uses a bias-based load balancing strategy, without designing any diffusion-specific routing mechanism or auxiliary routing objective function.
PixelMoE introduces Pixel-Space Hierarchical Flow Matching, which varies the spatial resolution along the denoising trajectory: early stages use low resolution with a large receptive field to capture global structure, and later stages use high resolution to refine details.
On the class-conditional ImageNet-1K benchmark, PixelMoE achieves FID 1.47 at 256×256 resolution and FID 1.53 at 512×512, without using representation alignment and perceptual losses.
Compared with the dense version, at equivalent active parameter counts, PixelMoE consistently outperforms the corresponding dense scheme across all model sizes.
Sparse scaling through conditional computation makes 'total parameter count' and 'active parameter count' two independent dimensions, where the former determines the upper bound of model expressiveness and the latter determines inference cost; PixelMoE leverages this decoupling to expand total capacity under a fixed active parameter count.
The value of PixelMoE lies in proving that scaling pixel-space diffusion models can reuse the MoE engineering experience and best practices already accumulated by the LLM community, including training infrastructure, routing strategies, load balancing schemes and parallelization implementations.
PixelMoE's FID results are competitive with current leading pixel-space and latent-space generative models.
All Facts (8)
PixelMoE proves that the Mixture-of-Experts (MoE) architecture, mature in the LLM field, can be directly transferred to pixel-space diffusion models without any diffusion-specific routing mechanism.
50%UnverifiedPixelMoE directly adopts the standard Token-Choice routing from LLMs, keeps one shared expert, and uses a bias-based load balancing strategy, without designing any diffusion-specific routing mechanism or auxiliary routing objective function.
50%UnverifiedPixelMoE introduces Pixel-Space Hierarchical Flow Matching, which varies the spatial resolution along the denoising trajectory: early stages use low resolution with a large receptive field to capture global structure, and later stages use high resolution to refine details.
50%UnverifiedOn the class-conditional ImageNet-1K benchmark, PixelMoE achieves FID 1.47 at 256×256 resolution and FID 1.53 at 512×512, without using representation alignment and perceptual losses.
50%UnverifiedCompared with the dense version, at equivalent active parameter counts, PixelMoE consistently outperforms the corresponding dense scheme across all model sizes.
50%UnverifiedSparse scaling through conditional computation makes 'total parameter count' and 'active parameter count' two independent dimensions, where the former determines the upper bound of model expressiveness and the latter determines inference cost; PixelMoE leverages this decoupling to expand total capacity under a fixed active parameter count.
50%UnverifiedThe value of PixelMoE lies in proving that scaling pixel-space diffusion models can reuse the MoE engineering experience and best practices already accumulated by the LLM community, including training infrastructure, routing strategies, load balancing schemes and parallelization implementations.
50%UnverifiedPixelMoE's FID results are competitive with current leading pixel-space and latent-space generative models.
50%