Structured Pruning
A model compression technique that removes entire functional units such as attention heads, MLP blocks, or channels. It can directly shrink matrix dimensions and thus accelerate inference; it is more practical than unstructured pruning but operates at a coarser granularity.
Timeline (last 90 days)
Existing structured pruning methods usually assign and rank importance weights at the MLP block or attention head level, with relatively coarse granularity, which loses the information about differences in importance among different neurons within an MLP block.
Structured pruning achieves hardware acceleration more easily in practice compared to unstructured pruning.
Osprey uses a structured pruning strategy to compress a pre-trained small language model into a shallow backbone network, then restores its language modeling capability through target-agnostic next-token pre-training.
Osprey employs a structured pruning strategy to build a shallow backbone network, and then recovers language modeling capability through target-agnostic next-token pre-training.
Structured pruning removes entire neurons, attention heads, or even entire Transformer blocks as units; the pruned model requires no special sparse hardware support and can run efficiently directly on standard GPUs/CPUs.
Models of the same size obtained by pruning from a larger model can often achieve comparable or even better performance with fewer training tokens.
All Facts (6)
Existing structured pruning methods usually assign and rank importance weights at the MLP block or attention head level, with relatively coarse granularity, which loses the information about differences in importance among different neurons within an MLP block.
50%UnverifiedStructured pruning achieves hardware acceleration more easily in practice compared to unstructured pruning.
50%UnverifiedOsprey uses a structured pruning strategy to compress a pre-trained small language model into a shallow backbone network, then restores its language modeling capability through target-agnostic next-token pre-training.
50%UnverifiedOsprey employs a structured pruning strategy to build a shallow backbone network, and then recovers language modeling capability through target-agnostic next-token pre-training.
50%UnverifiedStructured pruning removes entire neurons, attention heads, or even entire Transformer blocks as units; the pruned model requires no special sparse hardware support and can run efficiently directly on standard GPUs/CPUs.
50%UnverifiedModels of the same size obtained by pruning from a larger model can often achieve comparable or even better performance with fewer training tokens.
50%