Images Throw Off VLM Annotators Even When Told to Ignore Them — MIST Benchmark Exposes a Hidden Flaw in Multimodal Models

Using VLMs to Replace Human Annotators May Introduce Hidden Risks

A growing number of companies and research teams are turning to vision-language models (VLMs) to replace human annotators — the appeal is straightforward: lower cost, faster turnaround. But a paper by Nagham Omar and Loughton Pelt-Cohen throws cold water on this trend, revealing a serious and long-overlooked vulnerability in VLMs as annotation proxies.

The problem stems from how these systems are validated. The standard "surrogate annotation test" measures model-human agreement under a single, fixed set of conditions. But real-world annotation often involves irrelevant context sitting alongside the content being labeled — an accompanying image, a preceding message — that annotation guidelines explicitly instruct annotators to ignore. Humans can do this reliably. Models may not.

This vulnerability is more dangerous than it appears. Once a model passes a surrogate annotation test and gets deployed to generate labeled data, that data becomes the training and evaluation sets for downstream models. If irrelevant context causes systematic labeling errors, those errors propagate through the entire AI development pipeline — skewing model comparisons, corrupting research conclusions, and making late-stage corrections extremely costly.

Why Idiom Recognition Is the Perfect Test Case

Prior research has documented various biases in VLMs acting as judges: a preference for longer responses, a tendency to favor their own outputs, and scores that fluctuate with prompt wording and rating formats. But these have generally been treated as accuracy problems solvable through engineering, rather than as fundamental threats to the validity of the "model can replace human" claim itself.

The cleverness of this paper lies in its choice of task. The meaning of an idiom is determined entirely by its sentence context — what image appears alongside it is irrelevant. "Kick the bucket" means "to die," regardless of whether a bucket appears in the picture. And many idioms support both literal and figurative readings: "My grandfather kicked the bucket last winter" is figurative; "He kicked the bucket and spilled the water" is literal.

This means researchers can swap out the accompanying image without changing the sentence at all — a clean way to test whether models get distracted by irrelevant visuals. This is the inverse of how most multimodal benchmarks are designed.

Benchmarks like IRFL and AdMIRe treat the image itself as the object of classification — when the image changes, the correct answer changes too, making it impossible to isolate distraction from genuine task performance. Another dataset, ID10M-JAM, does perturb inputs without changing the correct label, but it perturbs text rather than images, and only measures accuracy rather than human-model agreement. This paper fills that gap: it specifically distinguishes models that can appropriately use context from models that get derailed by context they were told to ignore.

The surrogate annotation test is a methodological framework for validating whether an automated annotation system can substitute for human annotators. The core approach is to collect human labels and model labels on the same set of samples, then measure agreement using statistics like Cohen's Kappa or Fleiss' Kappa. If agreement exceeds a preset threshold, the model is deemed an acceptable human substitute. This framework is widely used in the NLP community, especially for large-scale annotation projects in areas like sentiment analysis and toxicity detection. However, it rests on a hidden assumption: that the input conditions during testing are identical to those during deployment. If real-world inputs include additional information not present at test time — such as images or preceding messages — the validity of the test results cannot be guaranteed. Fleiss' Kappa is a statistical measure of inter-rater agreement among multiple annotators, ranging from -1 to 1; values above 0.6 are generally considered a threshold for "good agreement."

MIST: Misleading Image Stress Test

To investigate this, the researchers built MIST (Misleading Image Stress Test): 200 English entries, each consisting of a sentence containing a potential idiom. The correct label is determined entirely by the sentence, and annotation instructions explicitly state that judgments should be based only on the sentence — not on any accompanying image.

Each sentence can be paired with three types of input: no image, an "aligned" image consistent with the sentence's meaning, or a "misleading" image inconsistent with it. Crucially, both types of images represent valid interpretations of the idiom — they just differ in how well they match the specific sentence. This design rules out the confound of images being wholly unrelated.

The test covers four conditions: figurative sentences paired with literal images (e.g., "hit the sack" paired with a picture of someone hitting a sandbag), figurative sentences with figurative images, literal sentences with literal images, and literal sentences with figurative images. Conditions two and three are aligned; conditions one and four are misleading. Each condition contains 50 samples.

The label scheme is also more nuanced than a simple binary, using four partially ordered levels: fully figurative, weakly figurative, figurative-literal coexistence, and fully literal. Judgment proceeds in two steps to capture variation that a binary classification would miss. Human annotations were collected by two groups of three fluent English speakers each; the two groups' Fleiss' Kappa scores were 0.65 and 0.57 respectively, with no significant difference.

Chain-of-Thought (CoT) prompting is a technique that asks a model to output intermediate reasoning steps before giving a final answer, introduced by Wei et al. in 2022 and quickly adopted as a standard tool for improving reasoning in large language models. The underlying assumption is that explicit step-by-step reasoning helps models decompose complex problems and reduces errors from jumping directly to conclusions. In annotation tasks, CoT has been seen as a way to make models follow rules more faithfully. However, this study found that even with CoT prompting, models still couldn't reliably ignore irrelevant images — suggesting that "making a model explain its reasoning" is not the same as "making a model respect annotation constraints." These are distinct capabilities. Few-shot prompting, by contrast, guides model output format and judgment criteria by embedding correct examples directly in the prompt, and is currently one of the most common prompt optimization strategies in practice.

The Core Findings That Upend Assumptions

The study evaluated 13 VLMs spanning five major families — GPT, Gemini, Gemma, Mistral, and Qwen — with parameter counts ranging from 3B to 32B. Seven of these passed the surrogate annotation test under at least one configuration. Each model was tested with four prompt types (zero-shot, few-shot, chain-of-thought, and few-shot + CoT), each in two variants ("explicitly told to ignore the image" vs. "no mention of the image"), producing approximately 52,000 labels in total.

The conclusions are deeply counterintuitive:

• Any image — regardless of its content — disrupts model judgment. Among the 7 models that passed the surrogate test, adding an aligned image caused 15.9% of labels to change; adding a misleading image caused 14.7% to change. The gap between the two is less than 5 percentage points. By contrast, the control condition of changing only the prompt wording produced just 8.8% label shifts — meaning that even explicitly instructing the model to ignore the image, the mere presence of an image causes more disruption than changing the prompt itself.

• The disruption doesn't pull judgments toward the image's meaning. Across all 13 models, only 37% of changed labels shifted toward the misleading image's interpretation — 63% moved in the opposite direction. Every model fell below the 50% random baseline. Images appear to randomly scramble model judgments rather than systematically guide them.

• Overall accuracy barely changes. Agreement with human labels was 54.4% without images, 53.7% with aligned images, and 54.7% with misleading images. Images change which samples the model gets right, not the total number — some samples flip from correct to incorrect while others flip from incorrect to correct, and the effects cancel out.

Furthermore, 71% of label changes did not cross the core figurative-literal boundary — they only shifted within the same category. This explains why conventional surrogate annotation tests miss the problem entirely: aggregate agreement metrics completely mask the sample-level volatility.

Understanding these results requires distinguishing two levels of "agreement": overall agreement rate (the proportion of all samples where the model matches humans) and sample-level stability (whether the model produces the same judgment for a given sample across different input conditions). Traditional surrogate tests only measure the former. This study shows the two can be completely decoupled — overall agreement stays around 54%, but about 15% of samples quietly flip from correct to incorrect when an image is added, while another 15% flip from incorrect to correct, with the two error groups canceling each other out and leaving aggregate metrics undisturbed. This phenomenon is known in statistics as cancellation error: invisible in aggregate metrics, but generating systematic noise in downstream tasks, since what ultimately enters the training set is the label for each individual sample — not the average agreement rate.

A Real Warning for the AI Development Pipeline

A noteworthy contrast: the 7 models that passed the surrogate test showed a 15.9% label change rate, while the 6 that failed showed 25.8%. Higher-performing models — those most likely to actually be deployed — are less susceptible to distraction, but still more affected than prompt changes alone. This indicates the problem is pervasive, not limited to weaker models.

The study's most important warning is this: when we say a model "can replace human annotators," we are describing its performance under a specific configuration with specific input conditions — not a stable, intrinsic property of the model. Once deployment conditions differ from test conditions (say, by including irrelevant images), that conclusion no longer holds.

The paper therefore recommends that model deployment reports must explicitly document the prompts used and whether any extraneous context was present — it is not enough to simply state that the model passed a surrogate annotation test. MIST is essentially an invariance test for annotation models: it uses cross-modal perturbations that don't change the correct answer to check stability. This approach can be extended to other annotation settings where text and images coexist, such as e-commerce product descriptions or social media content.

The researchers also acknowledge limitations: both image types were always related to the idiom, so fully irrelevant images were not tested; the partially ordered label structure means changes at different positions carry different weight; each human annotator only saw one condition, so there is no baseline for human-level label variation; all samples are English idioms; and model reasoning mode was disabled.

That said, observing the same effect consistently across 13 models from different families and parameter sizes is strong evidence that this is a general problem with VLMs as annotators. The dataset is publicly available on Hugging Face (Nagham/Mist-VLM-Judges), and practitioners can use it directly to assess the stability of their own annotation models. It also points to a new direction for VLM improvement: beyond enhancing cross-modal fusion, models need the ability to selectively ignore irrelevant modality information and strictly focus on the inputs specified by the task — a capability where current VLMs still have significant room to grow.