The POPE Benchmark's Yes/No Extraction Pitfall: VLM Hallucination Evaluation Confounded by Token Choice

A reproduction study of the POPE hallucination benchmark reveals a neglected evaluation trap: the way Yes/No answers are extracted from vision-language models (VLMs) can by itself cause F1 scores to drift by several points—a variable that has never been disclosed in most published work.

Starting Point: A Baseline 1.21 to 8.67 Points Too High

The researchers' original goal was to reproduce a recent hallucination-mitigation result. On LLaVA-1.5-7B, they measured an adversarial-split F1 of 82.21—1.21 to 8.67 points higher than the three adversarial baselines they planned to compare against (81.00, 79.30, 73.54).

Such a gap is not unusual in itself, but when the researchers opened the evaluation code, they found a reading choice that can independently shift F1—meaning the score difference may not stem from the method's merits but from "how the model's answers are read."

Core Finding: Two Token IDs That Were Never Generated

The Failure of Dual-Token Logit Reading

The conventional approach uses two vocabulary IDs to decide yes or no: 3582 for "yes" and 1217 for "no." The researchers ran statistics over 9,000 greedily decoded questions covering all three splits, and found that these two IDs were never generated by the model even once.

The model actually outputs " Yes" (token 3869) and " No" (token 1939). The reason is that SentencePiece appends a space byte after the prompt suffix "ASSISTANT:", so the model faces a surface form with a leading space rather than the bare word.

SentencePiece is a subword tokenization algorithm proposed by Google, widely adopted by models such as LLaMA and LLaVA. One of its key properties is treating spaces as ordinary characters (represented by ▁), rather than stripping them during preprocessing as WordPiece does. Thus, when the prompt ends with "ASSISTANT:", the tokenizer preserves a space byte after the colon, and the model's next-token prediction is effectively made on the surface " ▁Yes" with a leading space. This explains why the bare-word IDs (3582/1217) are never generated: those IDs correspond to the no-space versions in the model's vocabulary, while the actual decoding path lands on the space-prefixed variants. Understanding this is crucial for reproducing any logit-based Yes/No reading.

A 6.13-Point Adversarial Drift

After switching to an eight-token lookup table covering space-prefixed forms, LLaVA-1.5-7B achieved F1 scores of 0.8221, 0.8498, and 0.8713 on the adversarial, popular, and random splits, respectively. This single reading correction alone moved the adversarial split by 6.13 points—a magnitude exceeding the adversarial gains reported in five of the six papers the researchers surveyed.

This means that some of the "hallucination mitigation effects" claimed in the literature may stem, at least in substantial part, from differences in evaluation protocol rather than genuine improvements in model capability.

The F1 score is the harmonic mean of precision and recall, and is highly sensitive to the threshold and reading method for the positive class (yes). In binary benchmarks like POPE, yes typically represents "object present" and no represents "absent." If the lookup table covers only bare-word IDs while the model actually outputs space-prefixed variants, all of the model's yes predictions will be judged as "unrecognized," causing recall to plummet; conversely, if the lookup table covers space-prefixed variants but misses bare words, some no predictions may be misclassified as yes, inflating precision while depressing recall. The two mismatches differ in direction and magnitude, which is exactly the mechanism by which F1 can shift by several points. The eight-token lookup table adopted by the researchers covers both space-prefixed and non-prefixed yes/no variants (including case variants), thereby eliminating this reading ambiguity.

Protocol-Level Confounding: Validated Across Three Tokenizer Families

This is not an isolated issue for one model. The researchers conducted a four-reading audit on four other VLMs spanning three tokenizer families, concluding that the confounding is at the protocol level rather than the model level.

The most direct evidence: transplanting LLaVA-1.5's token IDs verbatim to two models with disjoint vocabularies caused F1 to drop to 0.00, whereas tokenizer-derived readings kept F1 stable between 0.80 and 0.92. A single mismatched evaluation detail is enough to render an entire benchmark result meaningless.

Tokenizer families refer to the subword segmentation schemes adopted by different models—common ones include SentencePiece (LLaMA family), BPE (GPT family), and WordPiece (BERT family). Vocabularies across families are disjoint, and the same semantic token has completely different IDs. For example, " Yes" is 3869 in LLaVA-1.5, but 3869 may correspond to a completely unrelated subword in another tokenizer family. Therefore, transplanting a model's hardcoded IDs directly to another model renders the reading completely invalid—this is why F1 drops to 0.00. The researchers emphasize "tokenizer-derived" readings: first encode "yes"/"no" and their space variants using the target model's tokenizer to dynamically obtain the ID set, then use it for logit comparison. This approach is portable across tokenizer families and represents the minimal specification that cross-model evaluation protocols should follow.

Why the Adversarial Split Resists Logit-Level Interventions

On the fixed baseline, the researchers tried nine inference-time correction methods, none of which surpassed the baseline. They further analyzed why the adversarial split is so stubborn.

A 3,000-question probe experiment yielded a counterintuitive result: incorrect predictions actually attracted more image attention than correct ones (Cohen's d = -0.46, p = 1.4×10⁻²⁴). This directly contradicts the core assumption of grounding-based correction methods—those methods assume the model answers incorrectly because it "didn't look at the image enough," but the data show the model looks more when it answers incorrectly.

An Unresolved Issue: Precision Matching

The researchers candidly acknowledge an open question: their reference runs were under 4-bit NF4 quantization, so they cannot determine how much of the reported gains would remain under conditions where both reading and precision are matched. This comparison cannot be concluded until the reading protocol and numerical precision are aligned simultaneously.

Implications for VLM Evaluation Practice

The value of this audit lies not in refuting any specific method, but in pointing out a systemic blind spot: most papers do not specify their Yes/No reading method, leaving readers unable to judge whether scores are comparable. For work relying on benchmarks like POPE to evaluate hallucination, at minimum two things should be done—disclose the token set used for reading and its source (hardcoded IDs or tokenizer-derived), and report inference precision.

In an era where benchmark scores are widely used as the basis for model selection, the transparency of evaluation protocols is as important as model capability itself.

One-line Summary

Mismatched Yes/No reading methods in the POPE benchmark can shift VLM hallucination F1 by over 6 points—a protocol-level evaluation flaw.

Paragraph Summary

While reproducing LLaVA-1.5-7B's performance on the POPE hallucination benchmark, the study found that the hardcoded yes/no token IDs (3582/1217) were never generated by the model across 9,000 questions—the model actually outputs space-prefixed " Yes" (3869) and " No" (1939), rooted in SentencePiece's handling of the space after the prompt suffix. After switching to a tokenizer-derived eight-token lookup table, the adversarial split's F1 shifted by 6.13 points, exceeding the adversarial gains reported in five of the six surveyed papers. This confounding was validated across four VLMs and three tokenizer families: transplanting hardcoded IDs across vocabularies dropped F1 to 0.00. A 3,000-question probe also found that incorrect predictions attracted more image attention (d=-0.46), contrary to grounding-correction assumptions. The authors call for disclosing the source of reading token sets and inference precision.