Optimization Faithfulness
A dimension of code optimization evaluation referring to whether the generated program genuinely contains the target optimization pattern at the abstract syntax tree level. Structural features of the transformation are detected through pattern matching or graph isomorphism algorithms, independent of hardware and input data.
Timeline (last 90 days)
PatternBench performs a dual evaluation of each generated program: runtime evaluation (measuring actual execution speed) and optimization fidelity analysis (verifying based on the AST whether the intended optimization pattern is implemented).
Optimization fidelity refers to whether the generated program truly contains the target optimization pattern at the abstract syntax tree (AST) level, and its technical implementation usually includes parsing the model output into an AST and then detecting the structural features of specific transformations through pattern matching or graph isomorphism algorithms.
Optimization fidelity analysis can only verify predefined transformation patterns and cannot discover optimizations invented by the model that are not listed in the benchmark; some semantically equivalent transformations may appear different on the AST, leading to missed detection.
Experiments found that optimization fidelity is strongly correlated with the achieved speedup and can serve as a reliable proxy metric for optimization capability.
All Facts (4)
PatternBench performs a dual evaluation of each generated program: runtime evaluation (measuring actual execution speed) and optimization fidelity analysis (verifying based on the AST whether the intended optimization pattern is implemented).
50%UnverifiedOptimization fidelity refers to whether the generated program truly contains the target optimization pattern at the abstract syntax tree (AST) level, and its technical implementation usually includes parsing the model output into an AST and then detecting the structural features of specific transformations through pattern matching or graph isomorphism algorithms.
50%UnverifiedOptimization fidelity analysis can only verify predefined transformation patterns and cannot discover optimizations invented by the model that are not listed in the benchmark; some semantically equivalent transformations may appear different on the AST, leading to missed detection.
50%UnverifiedExperiments found that optimization fidelity is strongly correlated with the achieved speedup and can serve as a reliable proxy metric for optimization capability.
50%