MIST
A benchmark dataset designed to test the robustness of vision-language models against distracting inputs. It evaluates whether models are misled by irrelevant images in idiom recognition tasks by introducing either aligned or misleading images, assessing the degree to which extraneous visual content influences annotation judgments.
Timeline (last 90 days)
Researchers constructed MIST (Misleading Image Stress Test), containing 200 English items, each of which is a sentence with a potentially idiomatic expression, where the correct label is determined entirely by the sentence.
The MIST test set covers four conditions: figurative-meaning sentences paired with literal-meaning images, figurative-meaning sentences paired with figurative-meaning images, literal-meaning sentences paired with literal-meaning images, and literal-meaning sentences paired with figurative-meaning images, with 50 samples per condition.
Human annotation was completed by two groups of three fluent English speakers each, with Fleiss' Kappa values of 0.65 and 0.57 respectively for the two groups, showing no significant difference.
7 out of the 13 evaluated VLMs passed the surrogate test under at least one configuration.
Among the 7 models that passed the test, adding aligned images led to a 15.9% label change, adding misleading images led to a 14.7% change, while the control group that only modified the prompt had just 8.8%.
Among all 13 models, only 37% of samples with label changes shifted toward the misleading image's meaning, while 63% went in the opposite direction, both below the 50% random baseline, indicating that images merely randomly disrupted model judgments rather than guiding them.
Overall accuracy remained almost unchanged: agreement with humans was 54.4% without images, 53.7% with aligned images, and 54.7% with misleading images.
71% of label changes did not cross the core boundary between figurative and literal meaning, but only made fine adjustments within the same category.
The 7 models that passed the surrogate test had a change rate of 15.9%, while the 6 models that did not pass had a rate as high as 25.8%, indicating that better-performing models are less susceptible to interference but still more so than the effect of prompt modification.
Each model was paired with four prompt types—zero-shot, few-shot, chain-of-thought, and few-shot plus chain-of-thought—each in two groups, producing approximately 52,000 labels in total.
3 more timeline events
All Facts (13)
Limitations of this study include: both types of images were related to idioms and completely irrelevant images were not tested, labels were partially ordered, each human annotator only saw one condition, all samples were English idioms, and the model reasoning mode was turned off.
50%UnverifiedThe MIST test set covers four conditions: figurative-meaning sentences paired with literal-meaning images, figurative-meaning sentences paired with figurative-meaning images, literal-meaning sentences paired with literal-meaning images, and literal-meaning sentences paired with figurative-meaning images, with 50 samples per condition.
50%UnverifiedHuman annotation was completed by two groups of three fluent English speakers each, with Fleiss' Kappa values of 0.65 and 0.57 respectively for the two groups, showing no significant difference.
50%Unverified7 out of the 13 evaluated VLMs passed the surrogate test under at least one configuration.
50%UnverifiedAmong the 7 models that passed the test, adding aligned images led to a 15.9% label change, adding misleading images led to a 14.7% change, while the control group that only modified the prompt had just 8.8%.
50%UnverifiedAmong all 13 models, only 37% of samples with label changes shifted toward the misleading image's meaning, while 63% went in the opposite direction, both below the 50% random baseline, indicating that images merely randomly disrupted model judgments rather than guiding them.
50%UnverifiedOverall accuracy remained almost unchanged: agreement with humans was 54.4% without images, 53.7% with aligned images, and 54.7% with misleading images.
50%Unverified71% of label changes did not cross the core boundary between figurative and literal meaning, but only made fine adjustments within the same category.
50%UnverifiedThe 7 models that passed the surrogate test had a change rate of 15.9%, while the 6 models that did not pass had a rate as high as 25.8%, indicating that better-performing models are less susceptible to interference but still more so than the effect of prompt modification.
50%UnverifiedEach model was paired with four prompt types—zero-shot, few-shot, chain-of-thought, and few-shot plus chain-of-thought—each in two groups, producing approximately 52,000 labels in total.
50%UnverifiedResearchers constructed MIST (Misleading Image Stress Test), containing 200 English items, each of which is a sentence with a potentially idiomatic expression, where the correct label is determined entirely by the sentence.
50%UnverifiedThe paper recommends that when releasing model deployment reports, the prompt used and whether any irrelevant context was added must be explicitly stated, rather than simply saying the model passed the surrogate test.
50%UnverifiedThe MIST dataset has been made publicly available on Hugging Face under the name Nagham/Mist-VLM-Judges.
50%