Referential Gap
A failure mode in multimodal reasoning in which linguistic references fail to bind correctly to the relevant visual entities, regions or locations, causing failures in state tracking. It is independent of visual perception capability.
Timeline (last 90 days)
The researchers define the failure mode in which models' language is not correctly bound to visual entities, regions, or locations during visual reasoning as the "reference gap."
The reference gap is a bottleneck independent of visual perception, explaining some errors previously misjudged as insufficient perception.
Numbering is effective for counting tasks because it naturally provides a distinction between "counted/uncounted"; however, for tasks requiring spatial relations or region localization, the same numbering scheme may introduce interference.
Constructing task-adapted visual reference can serve as a training-free visual skill to improve multimodal reasoning without fine-tuning the model or adding extra perception modules.
Open questions left by the research include how to automatically generate appropriate reference for arbitrary tasks and how to choose the granularity of reference.
All Facts (5)
The researchers define the failure mode in which models' language is not correctly bound to visual entities, regions, or locations during visual reasoning as the "reference gap."
50%UnverifiedThe reference gap is a bottleneck independent of visual perception, explaining some errors previously misjudged as insufficient perception.
50%UnverifiedNumbering is effective for counting tasks because it naturally provides a distinction between "counted/uncounted"; however, for tasks requiring spatial relations or region localization, the same numbering scheme may introduce interference.
50%UnverifiedConstructing task-adapted visual reference can serve as a training-free visual skill to improve multimodal reasoning without fine-tuning the model or adding extra perception modules.
50%UnverifiedOpen questions left by the research include how to automatically generate appropriate reference for arbitrary tasks and how to choose the granularity of reference.
50%