PerfReasoning
A benchmark specifically designed to evaluate large language models' ability to reason about hardware performance. It assesses LLM capabilities on performance modeling tasks such as chip architecture and dataflow mapping across two dimensions: question-answering inference and analytical performance model code generation.
Timeline (last 90 days)
PerfReasoning is a benchmark for evaluating LLMs' hardware performance reasoning capabilities, published via an arXiv paper.
PerfReasoning evaluates LLMs from two levels: as direct performance reasoners, and as generators of analytical performance model code.
As direct performance reasoners, models are required to compare different mapping schemes and predict off-chip memory traffic and buffer requirements given workload, architecture, and mapping specifications.
On reasoning-based question-answering tasks, the strongest closed-source models achieved accuracy exceeding 90%.
On performance model code construction tasks, GPT-5.6 Sol achieved a pass rate exceeding 80%.
On performance model code construction tasks, the average pass rate of all model configurations other than GPT-5.6 Sol was below 15%, with extremely large fluctuations across multiple runs.
LLMs can produce seemingly plausible architectural reasoning, but there is a significant gap in reliably building performance models.
The research team stated they will publicly release the PerfReasoning benchmark to support reproducible evaluation.
All Facts (8)
LLMs can produce seemingly plausible architectural reasoning, but there is a significant gap in reliably building performance models.
50%UnverifiedPerfReasoning evaluates LLMs from two levels: as direct performance reasoners, and as generators of analytical performance model code.
50%UnverifiedAs direct performance reasoners, models are required to compare different mapping schemes and predict off-chip memory traffic and buffer requirements given workload, architecture, and mapping specifications.
50%UnverifiedOn reasoning-based question-answering tasks, the strongest closed-source models achieved accuracy exceeding 90%.
50%UnverifiedOn performance model code construction tasks, GPT-5.6 Sol achieved a pass rate exceeding 80%.
50%UnverifiedOn performance model code construction tasks, the average pass rate of all model configurations other than GPT-5.6 Sol was below 15%, with extremely large fluctuations across multiple runs.
50%UnverifiedThe research team stated they will publicly release the PerfReasoning benchmark to support reproducible evaluation.
50%UnverifiedPerfReasoning is a benchmark for evaluating LLMs' hardware performance reasoning capabilities, published via an arXiv paper.
50%