ACE-Compression: A New Capability-Profiling Evaluation Framework for Compressed LLMs

Model compression enables large language models to be deployed in resource-constrained environments, but the industry's evaluation of compression effectiveness has long remained at the level of "does task accuracy drop?" This evaluation approach overlooks a key fact: compression perturbs the knowledge distributed across billions of weights, and a single score cannot reveal which underlying capabilities are preserved and which silently collapse. The ACE-Compression framework is a capability-centric evaluation solution proposed precisely for this blind spot—it doesn't ask "can the compressed model still answer correctly," but rather "which of its core capabilities remain intact."

Why traditional compression evaluation falls short

The standard approach is to measure the retained performance of a compressed model on benchmark tasks and compare it with the original model. This seems reasonable but in fact reduces model capability to a single scalar. The problem is that compression (whether quantization or pruning) affects the model's internal knowledge distribution non-uniformly: some capabilities may be nearly lossless, while others may degrade substantially—and these degradations may show up as only a one- or two-percentage-point fluctuation in the average score.

In other words, a compressed model that drops only 2 points on an aggregate leaderboard may have already lost 20 points on a specific sub-capability you care about. Task scores mask the redistribution of capability structure, and deployment decisions need to know exactly this structural change.

The two mainstream approaches in model compression—quantization and pruning—have fundamentally different mechanisms. Quantization maps high-precision floating-point weights (such as FP32) to low-precision representations (such as INT8 or INT4), reducing the number of storage bits per weight, usually without changing the network structure, but introducing numerical rounding error; pruning directly removes some weights or neurons (such as magnitude-based trimming), changing the network's sparse structure. Quantization error is typically distributed uniformly across all parameters, whereas pruning's destructiveness is more concentrated in the removed connections, so the two have different patterns of impact on knowledge distribution—which is one of the important reasons why quantization systematically outperforms pruning in capability retention in the experiments.

Four deployment-oriented capability dimensions

ACE-Compression breaks evaluation down into four dimensions directly relevant to real deployment scenarios:

Procedural reasoning and constraint execution: whether the model can follow multi-step instructions and satisfy format and logic constraints. This type of capability is critical in agent and workflow scenarios.

Knowledge anchoring and truthfulness: whether the model's factual knowledge is reliable and whether it becomes more prone to hallucination after compression.

Social cognition and emotional understanding: the model's grasp of intent, emotion, and social context directly affects the experience of conversational products.

Persona fidelity and style consistency: whether the compressed model can still maintain a given tone, persona, and expressive style.

These four dimensions cover the main deployment forms from tool-type invocation to anthropomorphic interaction, and they are precisely the parts that task average scores are worst at reflecting.

Experimental findings: capabilities are redistributed

The research team evaluated more than 35 compressed checkpoints across six open-weight models, covering both mainstream approaches of quantization and pruning. The core conclusion can be summed up in one sentence: compression does not uniformly weaken the model but redistributes the capability profile.

Specifically, coarse, local, surface-level capability signals tend to be preserved, while fine-grained, long-tail, context-dependent capabilities become fragile. This is consistent with intuition—compression tends to preserve high-frequency, salient patterns, while edge knowledge and complex reasoning chains are more easily damaged.

Another notable finding is that quantization systematically outperforms pruning in capability retention. But the authors emphasize that the practical significance of this gap depends entirely on which capability the application needs: if your scenario depends precisely on the type of capability that pruning damages more severely, then the average conclusion "quantization is better" is of no guidance value to you.

Practical implications for deployers and algorithm designers

The value of this framework lies in turning compression from a "black-box accuracy drop" into a measurable, interpretable intervention. For deployers, choosing a compression method should not rely solely on leaderboard average scores but on how well the application's required capability profile matches the post-compression profile. For algorithm designers, diagnostic signals from capability dimensions can guide the design of more targeted compression strategies to protect the capabilities that truly matter.

This is also the fundamental difference between capability-centric evaluation and task-centric evaluation: the former answers "can it be used in my scenario," while the latter only answers "how many points are left overall." As compressed models increasingly enter production environments, this fine-grained evaluation perspective will become ever more necessary.

One-line Summary

ACE-Compression proposes capability-centric evaluation of compressed models, revealing that compression does not uniformly weaken but redistributes capabilities.

Paragraph Summary

The ACE-Compression framework addresses the blind spot of traditional compression evaluation that focuses only on task accuracy by proposing evaluation of compressed models across four deployment-oriented capability dimensions: procedural reasoning and constraint execution, knowledge anchoring and truthfulness, social cognition and emotional understanding, and persona fidelity and style consistency. Experiments on six open-weight models and more than 35 compressed checkpoints show that compression does not uniformly weaken the model but redistributes the capability profile: surface-level, high-frequency capabilities are preserved, while fine-grained, long-tail, and context-dependent capabilities become fragile; quantization systematically outperforms pruning in capability retention, but the practical significance of the gap depends on the specific capabilities the application requires. The framework helps deployers choose compression methods based on capability profiles and guides algorithm designers in targeted protection of critical capabilities.