Layer Pruning
A model compression technique that removes entire decoder layers of a Transformer (self-attention plus feed-forward network) to reduce parameter count and computation. It usually requires a small amount of recovery fine-tuning.
Timeline (last 90 days)
D3D employs layer-wise progressive pruning to reduce the extra computational overhead of asynchronous packed drafting, with each layer pruned according to its impact on acceptance length.
The study systematically compared expert pruning, FFN pruning, and layer pruning on four categories of math and coding benchmarks.
Layer pruning directly deletes entire decoder layers in the Transformer, reducing parameter count and computation, but it may lose long-tail knowledge or specific language capabilities carried by the removed layers, and often requires a small amount of recovery fine-tuning.
All Facts (3)
D3D employs layer-wise progressive pruning to reduce the extra computational overhead of asynchronous packed drafting, with each layer pruned according to its impact on acceptance length.
50%UnverifiedThe study systematically compared expert pruning, FFN pruning, and layer pruning on four categories of math and coding benchmarks.
50%UnverifiedLayer pruning directly deletes entire decoder layers in the Transformer, reducing parameter count and computation, but it may lose long-tail knowledge or specific language capabilities carried by the removed layers, and often requires a small amount of recovery fine-tuning.
50%