Curvature-Aware Pruning with CACTUS: A New Method That Preserves Influence Functions

Pruning Is More Than Compression: Why Influence Functions Are Worth Preserving

Model pruning is usually treated as a means of compression for deployment, but traditional methods rarely care whether the compressed sparse model can still "understand" the influence of data. This study from an rss source proposes an interesting perspective: the degree to which influence functions are preserved may directly determine the downstream performance of the pruned model.

Influence functions measure the impact of a single training sample on model parameters, and they rely on gradient information of the loss function with respect to the parameters. If, after pruning, the internal gradients of the model are severely misaligned with those of the original dense model, then the pruned model will not only suffer from degraded predictive capability, but downstream tasks that depend on gradients, such as data selection, will also fail. The quantitative metric given in the study is intuitive: the per-sample gradient cosine similarity between the dense model and the compressed model. The lower this number is, the less consistent the gradient directions of the two models are, and the worse the fidelity of the influence functions.

The problem is that explicitly writing gradient alignment into the pruning objective is computationally extremely expensive. The objective function needs to consider both the activations of each layer and the gradients from backpropagation, and this kind of curvature-aware objective has previously lacked an efficient solver.

Influence Functions are a technique in statistics used to estimate the degree to which an individual training sample affects model parameters or prediction results. The core idea is: if a certain sample is removed and the model is retrained, how much will the model parameters change? In the context of deep learning, precisely computing influence functions requires solving for the product of the inverse Hessian matrix and the gradient vector, which is extremely costly. Therefore, in practice, the conjugate gradient method or Hessian-vector products are often used for approximation. The Koh & Liang method proposed by Koh and Liang in 2017 introduced influence functions into deep learning for identifying mislabeled samples and understanding model behavior. In recent years, influence functions have also been used for data selection, curriculum learning, and model debugging. The "preservation of influence functions" discussed in this article essentially refers to whether the pruned model can still maintain per-sample gradient directions close to those of the dense model, because the approximate computation of influence functions directly depends on gradients. If the gradient cosine similarity drops substantially, downstream tasks such as influence-function-based data selection and sample reweighting will fail.

CACTUS: Using ADMM to Solve the Curvature-Aware Pruning Objective

To address the above difficulties, the authors propose CACTUS, an optimizer based on ADMM (Alternating Direction Method of Multipliers). It supports two forms of weight transformation: pure pruning, and sparse plus low-rank decomposition. The latter preserves a low-rank correction term while pruning, in order to compensate for the removed information.

Compared with LLM-Surgeon, which targets the same objective function, CACTUS's acceleration effect is quite significant. Official data claims that when performing 50% pruning on Qwen3-4B, CACTUS reduces the gap between the C4 perplexity and that of the dense model by 45%, while increasing the compressed-dense per-sample gradient cosine similarity on WikiText-2 from 0.670 to 0.755. In terms of speed, even under significantly weaker hardware conditions, CACTUS is still more than 28 times faster than LLM-Surgeon. Compared with the "more than an order of magnitude" acceleration described in the original text, this number is more concrete.

From an engineering perspective, the reason ADMM is suitable for this type of problem is that it can decompose the coupled curvature objective into subproblems that can be solved alternately, avoiding expensive operations directly on the entire Hessian approximation. This choice of solution itself also explains why previous methods were difficult to implement: it is not that the objective was wrong, but that there was a lack of a scalable solution path.

ADMM (Alternating Direction Method of Multipliers) is an algorithmic framework for solving separable convex optimization problems. It was proposed by Gabay and Mercier in 1976 and has been widely used in large-scale optimization and distributed computing in recent years. Its core idea is to introduce auxiliary variables to decompose the original problem into multiple subproblems, alternately updating the primal variables, auxiliary variables, and dual variables, thereby avoiding directly handling coupling terms. ADMM is particularly suitable for scenarios where the objective function contains multiple nonsmooth regularization terms, such as L1 pruning constraints and low-rank constraints, because each subproblem often has a closed-form solution or can be solved efficiently. In model pruning, the curvature-aware objective involves coupling between each layer's activations and backward gradients, making direct optimization difficult, whereas ADMM can decouple the pruning constraints and curvature loss, alternately solving sparse projection and gradient alignment subproblems, thereby converging within an acceptable time. In addition, ADMM's convergence has theoretical guarantees for convex problems, and there is also extensive experimental support for its effectiveness in nonconvex problems.

The Practical Benefit of Preserving Influence Functions: Data Selection

If pruning really preserves influence functions, then a natural test is: use the sparse model to perform data selection, then train a dense model on the selected data and see whether the effect approaches that of using the dense model itself for data selection.

The experiments give an affirmative answer. On Llama-3.2-3B, fine-tuning a completely new dense model on the 5% of data selected by CACTUS achieves 49.44% accuracy on BIG-Bench Hard. By comparison, data selected by the ALPS method only reaches 47.22%, while directly using the dense model to select data gives 49.54%. The gap between CACTUS and dense-model data selection is less than 0.1 percentage points, yet it avoids the computational overhead required for selection using the dense model.

The significance of this result is that it turns "pruning preserves influence functions" from a theoretical metric into a quantifiable downstream benefit. The gradient cosine similarity increasing to 0.755 corresponds to data selection quality almost matching that of the dense model.

Combining with LoGRA: Fidelity Under Gradient Projection

The study also explores the integration of CACTUS with LoGRA. LoGRA is a recently proposed gradient projection strategy used to maintain the optimization direction on sparse structures. On Qwen3-4B, after pruning MLP layers to 2:4 sparsity and adding a rank-64 correction, the combination of CACTUS and LoGRA can still maintain 0.711 cosine alignment with the C4 gradients after dense LoGRA projection.

This number shows that even after introducing structured sparsity and low-rank compensation, the gradient direction is not severely damaged. For scenarios that require continued downstream fine-tuning or gradient-based operations, this fidelity is a prerequisite for usability.

2:4 sparsity is a structured sparsity pattern introduced by the NVIDIA Ampere architecture, requiring that at most 2 nonzero values be retained among every 4 consecutive weights. This pattern can achieve about 2x inference acceleration on GPUs through sparse tensor cores while maintaining model accuracy. Compared with fully unstructured pruning, 2:4 sparsity is more conducive to hardware acceleration, but its constraints are stronger and may cause greater disturbance to gradient directions. Low-rank correction (rank-64 correction) means additionally retaining a low-rank matrix product after pruning to compensate for lost information, that is, approximating the original weight matrix as a sparse matrix plus the product of two low-rank matrices. Rank-64 indicates that the rank of the low-rank part is 64. This value is related to the model's hidden layer dimension and usually needs to be adjusted according to the compression ratio and accuracy requirements. Low-rank correction can significantly restore the representational capacity lost due to structured pruning, but it also increases computational and storage overhead. CACTUS still maintains 0.711 cosine alignment under 2:4 sparsity plus rank-64 correction, indicating that the method can effectively coordinate structured pruning and gradient fidelity.

Multi-Source Corroboration and Current Limitations

It should be noted that the analysis in this article is mainly based on a paper abstract from an rss source. The core experimental data (45% reduction in perplexity gap, cosine similarity from 0.670 to 0.755, 28x speedup, BIG-Bench Hard accuracy comparison) all come from the original authors' own statements, and there has not yet been independent third-party reproduction or verification. The 0.711 cosine alignment in the LoGRA integration is also a single-source result.

From the perspective of methodological positioning, CACTUS addresses the specific bottleneck that "curvature-aware pruning objectives lack an efficient solver," and its contribution is concentrated at the optimizer level. Whether the acceleration brought by ADMM can be maintained on larger-scale models (such as the hundred-billion-parameter level), and whether influence functions are still preserved under more aggressive compression ratios beyond 2:4 sparsity, are directions not fully covered by the original paper. For teams concerned with the reusability of pruned models, CACTUS provides a path aimed at gradient fidelity and is worth validating first on medium-scale models.

One-line Summary

CACTUS efficiently solves the curvature-aware pruning objective with ADMM, preserving influence functions and bringing sparse-model data selection close to dense models.

Paragraph Summary

This article introduces CACTUS, an ADMM-based curvature-aware pruning optimizer designed to address the problem that traditional pruning ignores the preservation of influence functions. CACTUS efficiently solves a pruning objective that includes gradient alignment through the alternating direction method of multipliers, supporting both pure pruning and sparse plus low-rank decomposition. When pruning Qwen3-4B by 50%, it reduces the C4 perplexity gap by 45%, improves per-sample gradient cosine similarity from 0.670 to 0.755, and is more than 28 times faster than LLM-Surgeon. In downstream experiments, training a dense model on 5% of data selected by CACTUS reaches 49.44% accuracy on BIG-Bench Hard, close to 49.54% from dense-model selection. Combined with LoGRA under 2:4 sparsity plus rank-64 correction, it still maintains 0.711 gradient cosine alignment. The research data all come from the paper's own claims and still require independent reproduction; performance on large-scale models and under more aggressive compression ratios remains to be verified.