CompressionBench
A benchmark for evaluating whether coding agents can autonomously complete LLM compression tasks. It formulates compression as a terminal task, requiring the agent to deliver a runnable compressed model and inference code within a storage budget.
Timeline (last 90 days)
CompressionBench is a benchmark designed to study whether coding agents can autonomously complete LLM compression tasks.
CompressionBench formalizes compression as a terminal task: the agent receives a source checkpoint, storage constraints, an inference environment and a time budget, builds compressed weights and executable inference code, and autonomously controls algorithms and experiments, with the search guided by perplexity feedback, and the final model passes independent verification.
In compression benchmarks, BF16 is usually used as the uncompressed baseline model to compare the performance loss after compression.
Across six agents and six model-size settings, 29 of 36 submissions passed independent verification.
Selecting for each task the agent design with the lowest perplexity, average perplexity increased by 3.4% over BF16, and downstream accuracy was 77.33%, compared with 79.56% for BF16.
Under near-fixed storage, the agents reduced perplexity in 21 of 24 revisions, showing that they can use the feedback signal for effective search.
Full-model feedback reversed 17 of 79 previous local rankings, indicating that judging by local metrics alone can be misleading and that global verification is essential for correctly evaluating compression schemes.
Generation and inference tests exposed failure modes that likelihood alone cannot capture, suggesting that compressed models cannot be evaluated by perplexity alone.
The study identifies representation design, experimental judgment and capability verification as the core elements of agent-driven model compression.
All Facts (9)
CompressionBench is a benchmark designed to study whether coding agents can autonomously complete LLM compression tasks.
50%UnverifiedCompressionBench formalizes compression as a terminal task: the agent receives a source checkpoint, storage constraints, an inference environment and a time budget, builds compressed weights and executable inference code, and autonomously controls algorithms and experiments, with the search guided by perplexity feedback, and the final model passes independent verification.
50%UnverifiedIn compression benchmarks, BF16 is usually used as the uncompressed baseline model to compare the performance loss after compression.
50%UnverifiedAcross six agents and six model-size settings, 29 of 36 submissions passed independent verification.
50%UnverifiedSelecting for each task the agent design with the lowest perplexity, average perplexity increased by 3.4% over BF16, and downstream accuracy was 77.33%, compared with 79.56% for BF16.
50%UnverifiedUnder near-fixed storage, the agents reduced perplexity in 21 of 24 revisions, showing that they can use the feedback signal for effective search.
50%UnverifiedFull-model feedback reversed 17 of 79 previous local rankings, indicating that judging by local metrics alone can be misleading and that global verification is essential for correctly evaluating compression schemes.
50%UnverifiedGeneration and inference tests exposed failure modes that likelihood alone cannot capture, suggesting that compressed models cannot be evaluated by perplexity alone.
50%UnverifiedThe study identifies representation design, experimental judgment and capability verification as the core elements of agent-driven model compression.
50%