ConceptSparse Scaling
Sparse Scaling
A scaling strategy that uses conditional computation so that a model has a large number of parameters while only a small fraction is activated for each input, decoupling total parameter count from inference cost. MoE is its typical implementation.
Timeline (last 90 days)
Oct 14
Sparse scaling through conditional computation makes 'total parameter count' and 'active parameter count' two independent dimensions, where the former determines the upper bound of model expressiveness and the latter determines inference cost; PixelMoE leverages this decoupling to expand total capacity under a fixed active parameter count.
Unverified50%