PHYRD
An evaluation benchmark for theoretical physics reasoning ability, comprising two task types, Reproduce and Discover, with sub-question-level scoring criteria. It is maintained as a living benchmark with continuous updates.
Timeline (last 90 days)
The PHYRD benchmark is designed specifically for reasoning ability in theoretical physics and contains 63 Reproduce and 43 Discover questions.
PHYRD divides tasks into two types, Reproduce (deriving published theoretical results) and Discover (deriving new conclusions after modifying theoretical conditions), to distinguish memorized recitation from flexible reasoning.
All PHYRD tasks are constructed by FormulaForest and come with reference answers and sub-question-level rubrics.
Tests on five models show that whole-question accuracy on the Reproduce task ranges from 49.21% to 68.25%.
Tool use helps four models complete more Discover questions, but conceptual errors and missing arguments still persist.
PHYRD is designed as a "living benchmark" that is continuously updated to reflect the frontiers of theoretical physics research.
All Facts (6)
The PHYRD benchmark is designed specifically for reasoning ability in theoretical physics and contains 63 Reproduce and 43 Discover questions.
50%UnverifiedPHYRD divides tasks into two types, Reproduce (deriving published theoretical results) and Discover (deriving new conclusions after modifying theoretical conditions), to distinguish memorized recitation from flexible reasoning.
50%UnverifiedAll PHYRD tasks are constructed by FormulaForest and come with reference answers and sub-question-level rubrics.
50%UnverifiedTests on five models show that whole-question accuracy on the Reproduce task ranges from 49.21% to 68.25%.
50%UnverifiedTool use helps four models complete more Discover questions, but conceptual errors and missing arguments still persist.
50%UnverifiedPHYRD is designed as a "living benchmark" that is continuously updated to reflect the frontiers of theoretical physics research.
50%