TestHallVQA: A New Document-Level Reasoning Benchmark for Large Vision-Language Models Under Redundant Context

The evaluation of large vision-language models (LVLMs) on visual question answering (VQA) tasks has long been constrained by fundamental flaws in benchmark design. A new benchmark study called TestHallVQA aims to close this gap: it combines document-level scale, human exam-level difficulty, and controllable context redundancy injection to expose the hidden weaknesses of leading models in realistic, complex scenarios.

The Dilemma of Existing VQA Benchmarks

Existing flat-media VQA benchmarks tend to address isolated challenges in isolation, failing to capture the complexity of real-world use cases. Researchers point out that one class of benchmarks focuses on long-document understanding but offers limited reasoning depth — models only need to locate information within large volumes of text without performing complex logical computation. Another class emphasizes sophisticated visual reasoning but is constrained to single-page, noise-free clean environments.

This divide means that current evaluations can neither measure a model's ability to retrieve evidence from long documents nor test its reasoning robustness in environments filled with distracting information. Yet real-world documents — such as a scientific exam paper — naturally exhibit both characteristics simultaneously: long, multi-page, information-dense, and demanding rigorous computation and logical reasoning.

The Overlooked Problem: Interference from Irrelevant Visual Tokens

One theoretical contribution of this research is its systematic focus on performance degradation caused by irrelevant visual tokens. When a model processes multi-image documents, large amounts of visual content unrelated to the current question get encoded into the context, and these redundant tokens dilute the model's attention to critical evidence.

The researchers provide theoretical analysis proving that this interference leads to measurable performance degradation. Yet prior to this work, the field had rarely quantified this phenomenon systematically. In other words, models that "see more" don't necessarily "answer better" — redundant information is itself an attack surface, one that existing evaluation frameworks have largely ignored.

Core Design of TestHallVQA

TestHallVQA is a multi-image VQA benchmark designed to simultaneously embody three dimensions: document-level scale, human exam-level difficulty, and comprehensive task coverage. Drawn from scientific exam scenarios, it naturally integrates the dual challenge of long-document understanding and complex reasoning.

The most critical innovation is its ability to controllably inject multi-level context redundancy. Researchers can deliberately tune the amount and layering of irrelevant information within documents, allowing them to observe how model performance changes as a function of redundancy level under controlled conditions. This controllability transforms "the effect of redundancy on reasoning" from a vague intuition into a precisely measurable variable.

The F1-R² Metric: Measuring Both Reasoning and Retrieval Robustness

To complement this benchmark, the research team proposes a new evaluation metric, F1-R². Traditional accuracy or F1 scores only reflect how many questions a model answers correctly, but cannot distinguish whether the model is "genuinely reasoning" or "getting lucky in a clean environment."

F1-R² is designed to jointly quantify two capability dimensions: the model's computational reasoning ability and its evidence retrieval robustness under document-level redundancy. Through this metric, researchers can decompose whether a performance drop stems from insufficient reasoning capability or weak interference resistance, providing a more fine-grained perspective for diagnosing problems.

Experimental Findings and Research Value

The researchers conducted extensive experiments and analysis on mainstream LVLMs, revealing potential weaknesses across multiple dimensions. While the abstract does not present specific numerical results, the conclusions are clear: current top-tier models perform far below ideal when confronted with real-world tasks that simultaneously combine long-document scale and redundant interference.

The significance of this work lies not merely in proposing yet another harder leaderboard, but in transforming "the interference of redundant context on reasoning" — a long-neglected problem — into a reproducible, quantifiable evaluation framework. For researchers seeking to improve model robustness in real-world document scenarios, this provides concrete diagnostic tools and directions for improvement.

The associated dataset, code, and complete theoretical derivations have been open-sourced on GitHub (github.com/yqyu2317/TestHallVQA-benchmark) to facilitate community reproduction and further research.

Summary

The core value of TestHallVQA lies in integrating scattered evaluation challenges into a realistic scientific exam scenario, and in being the first to systematically quantify the interference effects of irrelevant visual tokens. Paired with the F1-R² metric, it provides a more realistic and discriminative benchmark for evaluating the document-level reasoning capabilities of LVLMs. For multimodal models striving to be practically reliable, the evaluation of this kind of "interference resistance" may well become a critical competitive dimension in the next phase of development.