[KongchangAI]
Conceptpopulation difficulty

Population-Level Difficulty

A population-level statistic of answering performance on a given set of problems, such as the average accuracy across multiple models or multiple samples and the average number of reasoning steps required. It is independent of any single output from a single model.

Timeline (last 90 days)

Nov 17

The association strength of perturbation inconsistency, semantic entropy, and solution length with overall difficulty is significantly higher than that of token-level uncertainty.

Unverified50%
Nov 17

Even when analyzing only the questions that the model answered correctly, the associations of perturbation inconsistency, semantic entropy, and solution length with overall difficulty remain significant.

Unverified50%
Nov 17

Overall difficulty usually refers to group-level answering performance on a given set of questions, such as the average accuracy of multiple models or multiple samples, and the average number of reasoning steps required.

Unverified50%

All Facts (3)

Source Articles