Population-Level Difficulty
A population-level statistic of answering performance on a given set of problems, such as the average accuracy across multiple models or multiple samples and the average number of reasoning steps required. It is independent of any single output from a single model.
Timeline (last 90 days)
The association strength of perturbation inconsistency, semantic entropy, and solution length with overall difficulty is significantly higher than that of token-level uncertainty.
Even when analyzing only the questions that the model answered correctly, the associations of perturbation inconsistency, semantic entropy, and solution length with overall difficulty remain significant.
Overall difficulty usually refers to group-level answering performance on a given set of questions, such as the average accuracy of multiple models or multiple samples, and the average number of reasoning steps required.
All Facts (3)
The association strength of perturbation inconsistency, semantic entropy, and solution length with overall difficulty is significantly higher than that of token-level uncertainty.
50%UnverifiedEven when analyzing only the questions that the model answered correctly, the associations of perturbation inconsistency, semantic entropy, and solution length with overall difficulty remain significant.
50%UnverifiedOverall difficulty usually refers to group-level answering performance on a given set of questions, such as the average accuracy of multiple models or multiple samples, and the average number of reasoning steps required.
50%