PatternBench
A benchmark specifically designed to evaluate the code optimization capabilities of large language models, focusing on compiler-beyond optimizations. It measures the model's genuine contribution through two dimensions: runtime evaluation and abstract syntax tree-based optimization faithfulness analysis.
Timeline (last 90 days)
PatternBench is a benchmark specifically designed to evaluate compiler-transcendent optimization, used to clarify the real contribution of LLMs in code optimization.
PatternBench covers seven categories of transformations (such as loop optimization and memory access pattern optimization).
PatternBench performs a dual evaluation of each generated program: runtime evaluation (measuring actual execution speed) and optimization fidelity analysis (verifying based on the AST whether the intended optimization pattern is implemented).
Experiments found that runtime speedup alone cannot fully characterize optimization capability: some models may achieve decent speedup but with very low fidelity, whereas conversely models with high fidelity often also bring actual speedup.
PatternBench chooses to evaluate models on residual optimization opportunities in order to exclude transformations the compiler can already cover, thereby measuring the model's optimization capability more purely.
The distribution of residual opportunities may vary with compiler version, target architecture, and optimization flags, so the benchmark needs to fix the compilation environment to ensure fair comparison.
All Facts (6)
PatternBench is a benchmark specifically designed to evaluate compiler-transcendent optimization, used to clarify the real contribution of LLMs in code optimization.
50%UnverifiedPatternBench covers seven categories of transformations (such as loop optimization and memory access pattern optimization).
50%UnverifiedPatternBench performs a dual evaluation of each generated program: runtime evaluation (measuring actual execution speed) and optimization fidelity analysis (verifying based on the AST whether the intended optimization pattern is implemented).
50%UnverifiedExperiments found that runtime speedup alone cannot fully characterize optimization capability: some models may achieve decent speedup but with very low fidelity, whereas conversely models with high fidelity often also bring actual speedup.
50%UnverifiedPatternBench chooses to evaluate models on residual optimization opportunities in order to exclude transformations the compiler can already cover, thereby measuring the model's optimization capability more purely.
50%UnverifiedThe distribution of residual opportunities may vary with compiler version, target architecture, and optimization flags, so the benchmark needs to fix the compilation environment to ensure fair comparison.
50%