Dense Model
A Transformer architecture in which all parameters are activated for every token during inference. In contrast to Mixture-of-Experts, its memory footprint and compute consumption are fixed and relatively high.
Timeline (last 90 days)
KV-Surgeon's experiments cover dense models and mixture-of-experts models with parameter sizes from 4 billion to 30 billion.
Traditional dense models use all parameters for every inference, so the complete weights must reside in memory or VRAM, otherwise every step of computation triggers a disk read.
The advantage of a dense architecture lies in more predictable behavior, more straightforward fine-tuning, and easier domain adaptation to achieve stable performance, at the cost of a relatively fixed inference computation load that cannot save computing power through sparse activation.
Traditional dense models activate all parameters when processing every token.
Compared to MoE architectures, dense models have more uniform gradient flow during training, are less prone to expert collapse, and exhibit more deterministic inference behavior.
Dense models have simpler inference logic and better hardware compatibility in local deployment scenarios, whereas MoE models, despite activating fewer parameters during inference, still require all weights to be loaded into memory, placing even higher demands on hardware.
All Facts (6)
KV-Surgeon's experiments cover dense models and mixture-of-experts models with parameter sizes from 4 billion to 30 billion.
50%UnverifiedTraditional dense models use all parameters for every inference, so the complete weights must reside in memory or VRAM, otherwise every step of computation triggers a disk read.
50%UnverifiedThe advantage of a dense architecture lies in more predictable behavior, more straightforward fine-tuning, and easier domain adaptation to achieve stable performance, at the cost of a relatively fixed inference computation load that cannot save computing power through sparse activation.
50%UnverifiedTraditional dense models activate all parameters when processing every token.
50%UnverifiedCompared to MoE architectures, dense models have more uniform gradient flow during training, are less prone to expert collapse, and exhibit more deterministic inference behavior.
50%UnverifiedDense models have simpler inference logic and better hardware compatibility in local deployment scenarios, whereas MoE models, despite activating fewer parameters during inference, still require all weights to be loaded into memory, placing even higher demands on hardware.
50%