POPE
A binary classification benchmark for evaluating hallucination in vision-language models. It assesses the degree of hallucination by asking whether objects exist in an image, and comprises three splits: adversarial, popular and random.
Timeline (last 90 days)
The Yes/No answer extraction method (token selection) of the POPE benchmark can by itself cause the F1 score of VLM hallucination evaluation to shift by several points, and this variable is undisclosed in most published work.
The adversarial split F1 measured on LLaVA-1.5-7B is 82.21, 1.21 to 8.67 points higher than the three adversarial baselines compared against (81.00, 79.30, 73.54).
After switching to an eight-token lookup table covering the space-prefixed forms, LLaVA-1.5-7B achieves F1 of 0.8221, 0.8498, and 0.8713 on the adversarial, popular, and random splits respectively; fixing the reading alone moves the adversarial split by 6.13 points.
The 6.13-point adversarial drift exceeds the adversarial gains reported in five of the six papers the researchers surveyed.
Of nine inference-time correction methods tried on a fixed baseline, none surpassed the baseline.
For work that relies on benchmarks such as POPE to evaluate hallucination, evaluation should disclose the token set used for Yes/No reading and its source (hardcoded IDs or tokenizer-derived), and report inference precision.
All Facts (6)
The Yes/No answer extraction method (token selection) of the POPE benchmark can by itself cause the F1 score of VLM hallucination evaluation to shift by several points, and this variable is undisclosed in most published work.
50%UnverifiedThe adversarial split F1 measured on LLaVA-1.5-7B is 82.21, 1.21 to 8.67 points higher than the three adversarial baselines compared against (81.00, 79.30, 73.54).
50%UnverifiedAfter switching to an eight-token lookup table covering the space-prefixed forms, LLaVA-1.5-7B achieves F1 of 0.8221, 0.8498, and 0.8713 on the adversarial, popular, and random splits respectively; fixing the reading alone moves the adversarial split by 6.13 points.
50%UnverifiedThe 6.13-point adversarial drift exceeds the adversarial gains reported in five of the six papers the researchers surveyed.
50%UnverifiedOf nine inference-time correction methods tried on a fixed baseline, none surpassed the baseline.
50%UnverifiedFor work that relies on benchmarks such as POPE to evaluate hallucination, evaluation should disclose the token set used for Yes/No reading and its source (hardcoded IDs or tokenizer-derived), and report inference precision.
50%