The "Reference Gap" in Multimodal Reasoning: What Can Visual Markers Fix?

Why Can Models "See" but Not "Figure It Out"?

Multimodal large language models have become quite proficient at recognizing visual content, but their reliability drops noticeably once they need to reason over visual information across multiple steps. A typical scenario is counting: the model can see all the objects in a scene, yet it loses track of "which ones have already been counted" during the process. Such errors are often attributed to insufficient perceptual ability, but a new study offers a different explanation—the problem lies in the reference step.

The researchers define this failure mode as the "reference gap": language is not correctly bound to the relevant visual entity, region, or location. In other words, the model isn't failing to see—it's failing to map "it" to "that one" in the image.

Visual Routines Theory: Shortening the Reference Distance with External Markers

The study is inspired by the theory of "visual routines." This theory holds that complex visual tasks can be decomposed into a series of primitive operations, and these operations depend on explicit reference to specific locations in a scene.

Following this line of thought, the researchers tested a simple hypothesis: if external references (such as numbering) are added to the image, can the model more reliably map language to visual objects? To this end, they built ReGap-Bench, a paired benchmark containing five categories of visual reasoning tasks.

The benchmark's design is key to controlling variables: for the same question, the scene, question, and answer are completely fixed, and the only thing that changes is the reference method. This way, any difference in accuracy can be attributed to reference rather than to perceptual difficulty or the question itself.

"Visual routines" were proposed by psychologist Shimon Ullman in the 1980s to explain how humans complete visual tasks that cannot be solved by a single fixation. Its core idea is that the visual system does not parse the entire image in one pass; instead, it invokes a set of composable "primitive operations"—such as moving the focus of attention, marking a location, and comparing features of two regions—executed in sequence to gradually build an understanding of the scene. The key to these operations is the ability to establish explicit reference to specific locations in the scene, i.e., "remember this spot first, then come back to it." This theory was later widely used to explain the cognitive mechanisms of visual search, line tracing, counting, and other tasks, and it also provides a conceptual framework for how computational models can organize multi-step visual reasoning.

Key Finding: It Only Helps When Matched—Mismatches Can Hurt

The experimental results lead to a clear conclusion:

• Reference methods that match the task can improve accuracy by up to 25%;
• When the reference method does not match the task, the gains are minimal and can even make performance worse.

This comparison shows that reference is not a universal enhancement that "helps as long as you add it," but rather an intervention that must be aligned with the task structure. Numbering works for counting tasks because it naturally provides a distinction between "counted/un counted"; but if the task requires spatial relations or region localization, the same numbering scheme may introduce interference.

Therefore, the reference gap is a bottleneck independent of visual perception—it explains part of the errors previously misjudged as "can't see clearly."

In multimodal models, "reference" is typically implemented through attention alignment between text tokens and visual tokens. The model needs to map linguistic expressions such as "the third one," "the one on the left," or "the ones already counted" to specific regions or objects in the image. When the correspondence between linguistic reference and visual features is ambiguous, attention may be distributed across multiple candidate objects, leading to failure in state tracking. The role of external numbering is to transform this implicit correspondence—which the model would otherwise have to infer on its own—into an explicit, directly matchable symbolic correspondence, thereby reducing the difficulty of binding language to vision. This also explains why the effect of reference is highly dependent on task structure: numbering provides ordinal information but cannot replace the expression of spatial relations or region boundaries.

A Training-Free "Visual Skill"

The more practical implication of this study is that constructing task-adapted visual references can itself serve as a training-free visual skill to improve multimodal reasoning.

No model fine-tuning is required, and no additional perception module is needed—only a change in how references are organized on the input side. This has direct value for real-world applications: in scenarios requiring multi-step visual reasoning, how visual information is presented may be just as important as the model itself.

Of course, the study also leaves open questions: how can appropriate references be automatically generated for arbitrary tasks? How should the granularity of references be chosen? These require follow-up work.

"Training-free" in such research usually means not updating model parameters and not introducing additional learnable modules, but only changing the organization of the input to influence model behavior. This falls into the same category of inference-time intervention as prompt engineering and in-context learning, but the difference is that it acts on the structure of visual input rather than merely adjusting textual instructions. Its advantages are plug-and-play use, low cost, and strong interpretability; its limitations are that the effect is bounded by the model's existing referencing ability and that reference schemes need to be manually designed for each task. For real-world applications requiring multi-step visual reasoning, this means that beyond model selection, the way input is presented is itself a design dimension worth optimizing.

Key Takeaways

The reference gap opens a crack in the familiar "perception vs. reasoning" dichotomy: sometimes the model's failure is neither that it cannot see nor that it cannot think, but that it has not anchored language to the correct visual object. The paired design of ReGap-Bench allows this variable to be measured in isolation, and the conclusion that "it only works when matched" reminds us that visual enhancement is not a universal patch but a design decision that must be aligned with task structure.