Capacity Tax

The cost in sparse models such as MoE that grows linearly with total stored parameter count, including weight storage, gradient synchronization, and optimizer state updates; even when the number of activated parameters stays constant, these overheads still scale with total parameter count.