dense scaling
A scaling approach in which all model parameters are used in every forward pass. The total number of parameters is proportional to inference computation, so model capacity and computational cost cannot be decoupled.
Timeline (last 90 days)
Existing pixel-space models (such as JiT) adopt a dense scaling strategy, where each increase in model capacity increases inference computation by the same amount, making it difficult to improve generation quality by scaling up parameters under a fixed inference budget.
Compared with the dense version, at equivalent active parameter counts, PixelMoE consistently outperforms the corresponding dense scheme across all model sizes.
All Facts (2)
Existing pixel-space models (such as JiT) adopt a dense scaling strategy, where each increase in model capacity increases inference computation by the same amount, making it difficult to improve generation quality by scaling up parameters under a fixed inference budget.
50%UnverifiedCompared with the dense version, at equivalent active parameter counts, PixelMoE consistently outperforms the corresponding dense scheme across all model sizes.
50%