Can Frontier Models Be Fooled by Visual Decoys? Elorian Breaks Through with Thinking in Images

Visual reasoning has always been the most delicate piece of the multimodal large model capability landscape. A model being able to "see" an image does not mean it can "understand" the image. A benchmark called "decoy counting" released by the Elorian team puts this problem directly on the table: when distracting visual elements are mixed into the scene, can frontier models still count correctly?

Decoy Counting: The Crack Between Seeing and Reasoning

The design of this benchmark is quite clever. It is not simply about having the model count how many objects are in an image, but about implanting "decoys" into counting scenarios—elements that look like targets but should not be counted. If the model relies only on shallow matching of visual features, it will easily count the decoys and give a wrong answer.

The results expose a clear gap: true visual perception and visual reasoning are not the same thing. Models can accurately describe the content of an image, yet lose points on steps that require suppressing interference and making judgments according to rules. This phenomenon of "seeing but not understanding" is precisely the key bottleneck for current multimodal models in being deployed in complex real-world scenarios.

Visual reasoning and visual perception have a clear distinction in cognitive science and artificial intelligence: perception refers to extracting features such as object category, position, and attributes from pixels, while reasoning requires performing judgments, comparisons, and deductions under rule constraints on top of perceptual results. Multimodal large models usually use a visual encoder (such as ViT) to map images into feature sequences, which are then connected to the language model through a projection layer, and the reasoning process occurs almost entirely in the text token space. This architecture naturally causes a break in the "perception-reasoning" information channel: before visual features enter the language model, they have already been compressed into a limited number of tokens, and fine-grained spatial relationships and information for distinguishing similar objects are easily lost. The decoy counting benchmark exploits exactly this weakness by implanting distractors highly similar to the targets into counting scenarios, testing whether the model has the ability to suppress irrelevant features and screen targets according to rules, rather than relying only on shallow visual matching.

Elorian Glimpse: Thinking with Images

To address this gap, Elorian's solution is Glimpse. Its core claim is to let models "think with images"—not to compress visual input into textual descriptions and then perform language reasoning, but to preserve and operate on visual representations themselves during the reasoning process.

The motivation behind this idea is straightforward: textual descriptions naturally lose information. When you turn an image into a sentence, the subtle differences between decoys and targets, spatial relationships, and visual similarities may all be smoothed away. Preserving the image channel in reasoning theoretically allows the model to lose less information on tasks such as counting, comparison, and spatial judgment.

According to Elorian's introduction, the goal of Glimpse is to build a bridge between perception and reasoning, turning "being able to see" into "being able to think clearly."

"Thinking with images" is a new paradigm in recent multimodal reasoning research. Representative approaches include inserting image cropping, region resampling, or visual token backfilling into the reasoning chain, enabling the model to revise judgments by "taking another look," much like humans do. Unlike the traditional two-stage pipeline of "image → textual description → language reasoning," such methods preserve the original visual representation in intermediate reasoning steps and allow the model to directly compare, count, or spatially locate image regions. Its technical prerequisites are that the features output by the visual encoder have sufficient spatial resolution and semantic fidelity, and that the language model can process text and visual tokens in an interleaved manner. The core claim of Elorian Glimpse belongs to this lineage, attempting to maintain the active participation of the image channel during reasoning so as to reduce information loss in the textualization process.

Why This Matters

Evaluation of multimodal models has long leaned toward descriptive tasks, such as image captioning and visual question answering. These tasks test whether the model can extract and restate information, but place limited demands on depth of reasoning. Benchmarks such as decoy counting fill in another piece of the puzzle: they force the model to make judgments under conditions with interference, which is closer to real-world applications.

Whether in document understanding, interface operation, or robot perception, the real world never provides clean input. Distractors, similar objects, and visual ambiguity are the norm. Whether a model can maintain stable judgment under these conditions directly determines whether it can move from demonstration to usability.

Elorian's attempt also points to a more fundamental question: whether the reasoning architecture of multimodal models should continue to be text-centric. If visual information is reduced to text before entering reasoning, then even stronger language reasoning ability cannot make up for the loss at the perception stage. Glimpse represents a reflection at the architectural level.

Multimodal evaluation has long been dominated by descriptive tasks such as COCO Caption and VQA. These benchmarks mainly measure the model's ability to extract and restate image content, and they are insufficient for examining multi-step reasoning and interference suppression. Visual reasoning benchmarks that have emerged in recent years have begun to introduce compositionality, counterfactuals, and distractor designs, for example mixing similar objects into counting tasks or adding mirror confusion to spatial relationship judgments. Such evaluations are closer to real application scenarios such as document understanding, GUI operation, and robot perception, because the inputs in these scenarios naturally contain a large number of irrelevant elements and visual ambiguity. The evolution of evaluation paradigms is therefore not only an update of measurement tools, but also reshapes the design direction of models in reverse: when evaluation reveals a systematic bias of "strong perception, weak reasoning," reflection at the architectural level becomes necessary.

Questions to Watch

Currently, publicly available information is still limited. The specific improvement of Glimpse on the decoy counting benchmark, its comparison results with other visual reasoning methods, and whether this approach can generalize to tasks beyond counting have not yet been fully disclosed.

One dimension worth watching is efficiency. Preserving visual representations in reasoning means higher computational overhead, and whether this cost is acceptable in actual deployment will affect the scope of application of the method.

Another issue is the generalizability of the evaluation itself. Decoy counting captures one pain point in visual reasoning, but the challenges of visual reasoning go far beyond counting. Spatial relationship reasoning, causal visual judgment, and multi-step visual planning all require their own evaluation tools. Elorian has made a start, but this road is still long.

Regardless, this work and the benchmark behind it remind us of one thing: progress in multimodal models cannot rely only on stacking parameters and feeding data; innovation in evaluation methods and reasoning paradigms is equally critical. The fact that visual decoys can fool frontier models shows that the standard by which we measure "understanding" itself also needs to be reexamined.

One-line Summary

Elorian releases the decoy counting benchmark, revealing the crack between perception and reasoning in multimodal models, and proposes the Glimpse approach of thinking with images.

Paragraph Summary

The Elorian team released a benchmark called "decoy counting," which implants distracting visual elements into counting scenarios to reveal a shortcoming of frontier multimodal models: they can describe an image accurately but lose points on steps requiring suppression of interference and rule-based judgment. To address this gap, Elorian proposes Glimpse, which advocates preserving and operating on visual representations during reasoning rather than first compressing images into textual descriptions, thereby reducing information loss. The article notes that multimodal evaluation has long emphasized descriptive tasks, and decoy counting fills in the puzzle piece of reasoning depth and is closer to the interference that is normal in real applications. This work also points to a deeper question: whether a text-centric reasoning architecture is sustainable. At present, Glimpse's specific gains, generalizability, and efficiency overhead have yet to be disclosed.