Perturbation Inconsistency
An uncertainty measurement method that applies slight semantics-preserving perturbations to the input and observes the stability of model outputs, reflecting the model's behavioral stability within a neighborhood of the input space.
Timeline (last 90 days)
The association strength of perturbation inconsistency, semantic entropy, and solution length with overall difficulty is significantly higher than that of token-level uncertainty.
Even when analyzing only the questions that the model answered correctly, the associations of perturbation inconsistency, semantic entropy, and solution length with overall difficulty remain significant.
Perturbation inconsistency measures uncertainty by applying slight, semantics-preserving perturbations to the input (such as paraphrasing, adding or removing irrelevant words, and adjusting word order) and observing whether the model's output remains stable.
Perturbation inconsistency aligns with findings in model robustness research.
All Facts (4)
The association strength of perturbation inconsistency, semantic entropy, and solution length with overall difficulty is significantly higher than that of token-level uncertainty.
50%UnverifiedEven when analyzing only the questions that the model answered correctly, the associations of perturbation inconsistency, semantic entropy, and solution length with overall difficulty remain significant.
50%UnverifiedPerturbation inconsistency measures uncertainty by applying slight, semantics-preserving perturbations to the input (such as paraphrasing, adding or removing irrelevant words, and adjusting word order) and observing whether the model's output remains stable.
50%UnverifiedPerturbation inconsistency aligns with findings in model robustness research.
50%