CACTUS
A curvature-aware model pruning method based on ADMM that preserves downstream task performance after pruning by retaining influence functions (per-sample gradient directions). It supports two weight transformations: pure pruning and sparse plus low-rank decomposition.
Timeline (last 90 days)
CACTUS is an optimizer based on ADMM (alternating direction method of multipliers) used to solve the curvature-aware pruning objective, supporting two weight transformation modes: pure pruning and sparse plus low-rank decomposition.
When doing 50% pruning on Qwen3-4B, CACTUS reduced the gap between C4 perplexity and the dense model by 45%.
When doing 50% pruning on Qwen3-4B, CACTUS raised the compressed-dense per-sample gradient cosine similarity on WikiText-2 from 0.670 to 0.755.
Under clearly weaker hardware conditions, CACTUS is still more than 28 times faster than LLM-Surgeon.
On Llama-3.2-3B, fine-tuning a brand-new dense model with the 5% of data selected by CACTUS reached 49.44% accuracy on BIG-Bench Hard; selecting data directly with the dense model gave 49.54%; data selected by the ALPS method could only reach 47.22%.
On Qwen3-4B, after pruning MLP layers to 2:4 sparsity and adding rank-64 correction, CACTUS combined with LoGRA still maintains 0.711 cosine alignment with the C4 gradients of dense LoGRA after projection.
CACTUS addresses the specific bottleneck that the curvature-aware pruning objective lacks an efficient solver, and its contribution is concentrated at the optimizer level.
The core experimental data in this article all come from the original text's own account, and there is not yet independent third-party reproduction and verification.
Whether the speedup brought by ADMM can be maintained on larger-scale models (such as the hundred-billion level), and whether influence functions are still preserved under more aggressive compression ratios beyond 2:4 sparsity, are directions not fully covered by the original text.
All Facts (9)
CACTUS is an optimizer based on ADMM (alternating direction method of multipliers) used to solve the curvature-aware pruning objective, supporting two weight transformation modes: pure pruning and sparse plus low-rank decomposition.
50%UnverifiedWhen doing 50% pruning on Qwen3-4B, CACTUS reduced the gap between C4 perplexity and the dense model by 45%.
50%UnverifiedWhen doing 50% pruning on Qwen3-4B, CACTUS raised the compressed-dense per-sample gradient cosine similarity on WikiText-2 from 0.670 to 0.755.
50%UnverifiedUnder clearly weaker hardware conditions, CACTUS is still more than 28 times faster than LLM-Surgeon.
50%UnverifiedOn Llama-3.2-3B, fine-tuning a brand-new dense model with the 5% of data selected by CACTUS reached 49.44% accuracy on BIG-Bench Hard; selecting data directly with the dense model gave 49.54%; data selected by the ALPS method could only reach 47.22%.
50%UnverifiedOn Qwen3-4B, after pruning MLP layers to 2:4 sparsity and adding rank-64 correction, CACTUS combined with LoGRA still maintains 0.711 cosine alignment with the C4 gradients of dense LoGRA after projection.
50%UnverifiedCACTUS addresses the specific bottleneck that the curvature-aware pruning objective lacks an efficient solver, and its contribution is concentrated at the optimizer level.
50%UnverifiedThe core experimental data in this article all come from the original text's own account, and there is not yet independent third-party reproduction and verification.
50%UnverifiedWhether the speedup brought by ADMM can be maintained on larger-scale models (such as the hundred-billion level), and whether influence functions are still preserved under more aggressive compression ratios beyond 2:4 sparsity, are directions not fully covered by the original text.
50%