TestHallVQA
TestHallVQA is a multi-image visual question answering benchmark for large vision-language models. It combines document-scale inputs, human exam-level difficulty, and controllable injection of contextual redundancy to systematically quantify the distraction effect of irrelevant visual tokens, accompanied by the F1-R² evaluation metric.
Timeline (last 90 days)
TestHallVQA is a multi-image VQA benchmark combining document-level scale, human exam-level difficulty, and controllable contextual redundancy injection.
TestHallVQA is drawn from scientific examination scenarios.
The research team proposed a new evaluation metric, F1-R², for jointly quantifying computational reasoning ability and evidence retrieval robustness under document-level redundancy.
Prior to this study, the academic community rarely conducted systematic quantification of performance degradation caused by irrelevant visual tokens.
The researchers conducted experiments on mainstream LVLMs, and the results show that current top models perform far below ideal levels when facing tasks that simultaneously involve long-document scale and redundancy interference.
The dataset, code, and theoretical derivations related to TestHallVQA have been open-sourced on GitHub at github.com/yqyu2317/TestHallVQA-benchmark.
All Facts (6)
TestHallVQA is a multi-image VQA benchmark combining document-level scale, human exam-level difficulty, and controllable contextual redundancy injection.
50%UnverifiedTestHallVQA is drawn from scientific examination scenarios.
50%UnverifiedThe research team proposed a new evaluation metric, F1-R², for jointly quantifying computational reasoning ability and evidence retrieval robustness under document-level redundancy.
50%UnverifiedPrior to this study, the academic community rarely conducted systematic quantification of performance degradation caused by irrelevant visual tokens.
50%UnverifiedThe researchers conducted experiments on mainstream LVLMs, and the results show that current top models perform far below ideal levels when facing tasks that simultaneously involve long-document scale and redundancy interference.
50%UnverifiedThe dataset, code, and theoretical derivations related to TestHallVQA have been open-sourced on GitHub at github.com/yqyu2317/TestHallVQA-benchmark.
50%