DeepSeek's Mysteriously Retracted Paper: Breaking AI's Multimodal Reasoning Bottleneck with Visual Primitives

A Paper That Stayed Online for Just 4 Hours

In the AI research community, DeepSeek has always been known for its steadiness—but this time, a dramatic scene unfolded. According to analysis from overseas AI bloggers, shortly after the release of DeepSeek v4, its researchers publicly teased a new vision-related project on social media. The paper and code repository were subsequently officially published—yet just 4 hours later, they were mysteriously taken down.

Fortunately, someone had saved a copy of this paper titled Thinking with Visual Primitives in advance, allowing the outside world to glimpse its contents. This is not only the formal realization of the multimodal direction DeepSeek promised to tackle in its v4 conclusions, but also proposes an entirely new perspective on visual reasoning.

Why the retraction after 4 hours? The blogger speculates one possible reason is that the team wanted to supplement more ablation experiments on the trade-off between the "perception gap" and the "reference gap"—because while the current version proves that visual primitives can make reasoning more efficient, it hasn't yet fully answered one key question: before better referencing mechanisms take effect, just how much can visual detail actually be compressed?

The Development Background of Multimodal Large Models

To understand the significance of this paper, one must first grasp the development trajectory of Multimodal Large Language Models (MLLMs). Since OpenAI released CLIP in 2021 and Google released Flamingo in 2022, multimodal models have undergone rapid iteration.

CLIP (Contrastive Language-Image Pre-training) maps images and text into the same semantic space through contrastive learning, achieving large-scale image-text alignment pretraining for the first time and establishing the foundational paradigm of modern multimodal models. Its core mechanism is: images and their corresponding text are respectively mapped into high-dimensional vectors through a visual encoder and a text encoder, with the training objective being to make matching image-text pairs closest in vector space and mismatched pairs farthest apart. CLIP was trained on 400 million image-text pairs, and its zero-shot transfer capability demonstrated that large-scale contrastive pretraining can learn general-purpose visual semantic representations—a finding that profoundly influenced the design choices of all subsequent multimodal architectures.

Flamingo introduced the Cross-Attention mechanism, allowing language models to dynamically query visual features when generating each token, pioneering the mainstream approach of "visually conditioned language generation." Unlike CLIP's static alignment, Flamingo's cross-attention layers use text tokens as Query and visual tokens as Key and Value, enabling the language model to constantly "look back" at different regions of the image during generation, achieving finer-grained visual-language interaction.

LLaVA (Large Language and Vision Assistant, 2023), as a milestone open-source architecture, proved that with just a linear projection layer to map ViT-output visual tokens into the language model's embedding space, combined with instruction fine-tuning data, one could obtain powerful multimodal conversational capabilities—this minimalist three-stage paradigm of "visual encoder + projection layer + LLM" dramatically reduced training costs compared to Flamingo's cross-attention scheme, while also avoiding the optimization difficulties that might arise from deeply embedding the visual module into the language model's interior. The LLaVA paradigm was inherited by numerous subsequent works such as InternVL and Qwen-VL, becoming the de facto industry standard.

Since then, closed-source models like GPT-4V, Gemini, and Claude 3, as well as open-source models like the LLaVA series, InternVL, and Qwen-VL, have emerged one after another, driving substantial improvements in perceptual capabilities. However, as perceptual capabilities approach saturation, deficiencies in visual reasoning capabilities have gradually become a new research focus—that is, models can see images clearly but cannot reliably reference what they see within complex reasoning chains. This is precisely the core problem DeepSeek's paper attempts to address head-on.

The Real Bottleneck: Not "Can't See," but "Can't Say Precisely"

If you're familiar with multimodal research, you may have had this feeling: the model can actually "see" every detail in an image, yet consistently makes mistakes when reasoning.

In the past, multimodal systems primarily tackled the perception gap—that is, the model couldn't see enough detail—with solutions including increasing resolution, dynamic cropping, and adding more image tokens. Research on the perception gap spawned a series of technical approaches: AnyRes (arbitrary resolution) processing handles high-resolution images by splitting them into multiple sub-images that are encoded separately and then concatenated with global features; dynamic ViT resolution adaptively adjusts the patch grid according to the input image's aspect ratio, avoiding geometric distortion caused by forced scaling. The core logic of this direction is: give the model more and higher-quality visual information, and reasoning will naturally improve.

But today, the real bottleneck has shifted to the reference gap: the model has clearly already seen the detail, yet cannot reliably "point back" to the exact visual entity it's discussing during reasoning. The essence of the reference gap is the manifestation of the "coreference resolution" problem in visual reasoning—the NLP field discovered long ago that when pronouns or ambiguous noun phrases span multiple reasoning steps, models are prone to reference drift. Coreference resolution is one of NLP's oldest challenges; classic methods like the Hobbs algorithm existed back in the rule-based era, and although neural networks in the deep learning era dramatically improved accuracy, coreference chains in multi-step cross-sentence reasoning remain a recognized fragile link. In visual scenarios, spatial overlap, appearance similarity, and occlusion effects further amplify this problem, causing models to easily "swap" the originally locked target for another visually similar object during multi-step reasoning. When a model needs to reference in reasoning step 5 a specific object it "saw" in step 1, the ambiguity of language description causes reference failure—highly analogous to how humans solving math problems on paper are prone to errors if they cannot annotate intermediate variables.

The essence of the problem is: language itself is not a sufficiently precise interface for visual reasoning. When we say "the bear on the left" or "the object behind the red cube," it sounds precise, but images are full of overlapping objects, dense layouts, partial occlusions, and vast ambiguous regions. When you ask the model to count all the people in a crowd, or navigate out of a maze, after a few reasoning steps it easily loses track of the object or path it's following—because the mechanism for tracking positions within an image simply doesn't exist. Even though the model has already seen the image, its chain of thought is still written in language, which leaves ample openings for hallucination, repeated counting, missed objects, and confused spatial relationships.

The Core Idea: Let the Model Think While "Pointing at the Image"

DeepSeek's solution is surprisingly direct: rather than forcing the model to reason only with text, let it actually point at the image while thinking.

Instead of describing in language "there's a bear on the ground," the model can say "there's a bear here," then output a bounding box framing that bear. Faced with a maze, it can directly output a string of coordinates outlining the actual path through the image.

Specifically, DeepSeek uses two types of visual primitives:

• Bounding box: used when the model needs to reference objects, suitable for counting and object localization;
• Point: used when the model needs to reason about trajectories, suitable for navigation-type and topological reasoning tasks.

These primitives are the medium of the model's internal reasoning, equivalent to the visual counterpart of "words" in a Chain of Thought (CoT). CoT was proposed by Google Research in 2022, and its theoretical foundation comes from the concept of "Working Memory Externalization" in cognitive science: when solving complex problems, humans use external tools (paper and pen, whiteboards, scratch paper) to solidify intermediate states, reducing the burden on the brain's working memory—psychological research shows that human working memory capacity is roughly 7±2 information chunks, and externalizing intermediate steps can break through this capacity bottleneck to handle reasoning chains of arbitrary length. Research has found that CoT's effectiveness heavily depends on the "precision" of intermediate state representations—CoT works best when the intermediate quantities involved can be precisely expressed symbolically (such as mathematical formulas); when intermediate quantities can only be described in vague natural language (such as spatial positions), CoT easily introduces cumulative errors. Visual primitives are precisely the spatial extension of this insight: replacing language descriptions with coordinates elevates the representational precision of "spatial intermediate states" from word granularity to pixel granularity, and the precision of coordinates fundamentally eliminates the ambiguity that inevitably arises when language describes spatial positions. It is for this reason that the paper's title is "Thinking with" rather than "Grounding with" visual primitives—this wording difference precisely highlights the paper's ambition.

Interestingly, this is no toy-level experiment. The team fine-tuned directly on the latest DeepSeek v4 flash (a 284-billion-parameter MoE model with 13 billion activated parameters) and developed its own visual encoder, DeepSeek ViT, alongside it. Mixture of Experts (MoE) is a conditional computation architecture: the network contains multiple parallel "expert" sub-networks, and during each forward pass a lightweight routing network (Router) dynamically selects which experts to activate based on the input, typically activating only the Top-K. Although DeepSeek v4 has 284 billion total parameters, each forward pass activates only about 13 billion parameters, achieving computational efficiency close to 1/20 that of a dense model of the same scale. The property of MoE that parameter scaling doesn't increase inference cost makes it the architecture of choice for ultra-large-scale models. Performing visual fine-tuning directly on a base model of such scale indicates the team believes there's no need to specially design visual perception architectures—strong visual reasoning capabilities can be achieved simply by layering a visual interface onto a general-purpose reasoning model.

Aggressive Image Compression: Was the Bottleneck Never Pixels?

The paper's architecture is similar to LLaVA: the image enters a ViT, the language instruction passes through a tokenizer, and the visual tokens and text tokens are concatenated before being fed into the LLM, with no cross-attention. What's truly interesting is its aggressive compression of images.

The problem of visual token redundancy is one of the core topics in multimodal model efficiency research. ViT (Vision Transformer) splits images into fixed-size patches and generates an independent token for each patch, but image signals have far stronger spatial local correlation than text—adjacent patches often share large amounts of semantic information, and this redundancy is especially pronounced in natural scene images (background regions, sky, ground, and other low-information-density areas consume a large portion of the token budget). Researchers have found that a considerable proportion of ViT-generated visual tokens contribute very little to the final task, giving rise to a batch of compression methods such as TokenPacker, LLaVA-PruMerge, and FastV. Leveraging its new attention mechanism CSA (Compressed Sparse Attention), DeepSeek's compression pipeline works as follows: the image is first split into 14×14 patches, then after the ViT a 3×3 spatial token compression is applied (merging every 9 adjacent patches into 1 visual token); after tokens enter the LLM, CSA further compresses every 4 visual tokens into 1 KV cache entry. CSA's design directly aggregates the KV cache of visual tokens at the attention computation level, rather than pruning at the token sequence level. Its advantage is that the model can achieve adaptive merging of local semantics while preserving global positional encodings, without introducing the training instability caused by an additional token selection network. This is similar to the compression approach of MQA (Multi-Query Attention), which has all attention heads share one set of KV, and GQA (Grouped-Query Attention), which has grouped heads share KV, but CSA is specifically optimized for the spatial locality of visual tokens, allowing adaptive merging within semantically similar spatial neighborhoods rather than forced uniform grouping.

Take a 756×756 image as an example (about 571,000 pixels): the ViT first converts it into 2916 patch tokens, which become 324 visual tokens after 3×3 compression, and after CSA compression, only 81 visual KV cache entries remain. From raw pixels to the final KV cache, the compression ratio is about 7056x, using 36x fewer KV cache entries than an ordinary LLaVA-style projection scheme.

This design clearly runs counter to the "perception gap" approach that past multimodal models focused on. The blogger thus raises an intriguing point: perhaps the bottleneck was never how many pixels you see, but whether your visual reasoning can truly reason things through. When a model fails, is it because there wasn't enough visual information, or because there was information but it couldn't be reliably referenced during reasoning? This may be precisely what the paper hasn't fully answered, and possibly one of the reasons it was retracted.

Targeted Training for Four Categories of Tasks

The team constructed training data for four categories of tasks, targeting the different failure modes of pure-language visual reasoning. All coordinates are normalized to 0-999, independent of resolution.

Counting: Detect-Localize-Filter-Count

Models typically fail at counting because they cannot maintain object identity. Therefore, the model is trained to first recognize intent, use boxes to localize all candidate objects, and then count from the localized set. Coarse-grained counting (such as "how many men are in the image") uses batch localization, framing all relevant objects at once and then summing to avoid duplicate enumeration; fine-grained counting (such as "how many bears are on the ground") first frames all bears, then filters those meeting the condition—bears climbing trees are excluded, while those standing on rocks or dirt are counted.

Spatial Reasoning: Real Scenes + Synthetic Scenes

The team used both GQA real scenes and CLEVR-style synthetic 3D scenes. GQA (Graph Question Answering) is a visual question answering benchmark released by Stanford University, characterized by generating structured questions based on Scene Graphs, with each question having a corresponding Functional Program annotating the reasoning steps, making it suitable for testing multi-hop visual reasoning capabilities. CLEVR (Compositional Language and Elementary Visual Reasoning) was proposed by MIT, using procedurally generated 3D geometric scenes where each object has precise 3D coordinates, attribute labels, and object IDs, enabling the generation of multi-hop questions where each step maps to a specific object, and can even project 3D coordinates back into 2D boxes to supervise the entire reasoning path, not just the final answer. Real-scene relationships are closer to reality but messier with incomplete annotations; CLEVR, though visually simple, provides supervision signals for intermediate reasoning steps through its procedurally generated precise ground truth. Their complementarity allows the training data to balance both distribution diversity and annotation precision.

Topological Reasoning: Mazes and Path Tracing

A maze is not an object detection problem but a connectivity problem. Analogizing maze solving to DFS (Depth-First Search) is not only an algorithmic metaphor but also reveals a deep isomorphism between visual reasoning and symbolic reasoning: DFS's core data structures are an explicitly maintained "stack of nodes to visit" and a "set of visited nodes," both of which can only be implicitly maintained through language description in pure-language reasoning, making state confusion highly likely; but when the model outputs a sequence of coordinate points, it is equivalent to externalizing the "visited path," and at each step the model only needs to judge the neighborhood connectivity of the current coordinate, without tracking global state in language space. In graph theory, a maze is essentially a shortest-path problem on a sparse undirected graph, and the correspondence between DFS's explicit stack structure and the sequence of visual primitive points reveals the information-theoretic equivalence between "spatial state externalization" and "algorithmic state externalization."

The model uses points as reasoning primitives, identifies the start and end points, and then outputs an exploration trajectory similar to DFS. The team even created "unsolvable mazes"—placing walls in the middle of the correct path of a solvable maze, forcing the model to genuinely reason about connectivity rather than pattern-match. This is a classic testing method for distinguishing "genuine reasoning" from "pattern memorization," entirely consistent with the approach in competitive programming of using extreme edge cases to verify program correctness.

Path tracing is the hardest challenge: the model must trace along a winding curve from one point to another. When two lines cross, humans continue tracing the same curve based on local geometric continuity, which is very difficult for a pure-language model to describe precisely. The model's solution is to output a series of points along the curve as waypoints, with adaptive density—few points for straight segments, more points at curves or dense intersections, effectively teaching the model to "slow down" when visual reasoning becomes difficult. This adaptive sampling density behavior is an emergent phenomenon: the training objective does not explicitly require density variation, but the model spontaneously learned the strategy that "difficult regions require finer-grained representation" during reinforcement learning, highly consistent with the intuitive human behavior of annotating more detail at complex intersections when drawing route maps.

Training Strategy and Benchmark Performance

The team built a total of 604,000 data samples (including 460,000 maze navigation, 125,000 path tracing, etc.), first separately training two SFT models—one specializing in boxes, the other in points—the early separation being to prevent the model from confusing which primitive format to use.

Subsequently, GRPO was used to perform reinforcement learning on each expert model. GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm proposed by the DeepSeek team, an improved variant of PPO (Proximal Policy Optimization). PPO is the mainstream policy gradient algorithm proposed by OpenAI in 2017, whose core is limiting the magnitude of policy updates through a clipping mechanism to prevent training instability caused by overly large updates; however, PPO requires training both a policy network (Actor) and a value network (Critic) simultaneously, and the value network's parameter count is usually comparable to that of the policy network, nearly doubling training costs, while the value network's estimation errors introduce additional variance—especially prominent in scenarios like language models with billions of parameters. GRPO's core innovation is replacing the value network with "intra-group relative ranking"—sampling N different outputs for the same question, using the group's average reward as the baseline, with each output's advantage equal to its reward minus the group average reward, then normalized. This design completely eliminates the value network while naturally correcting the absolute bias in reward scale through intra-group comparison, making the relative ranking information of the reward signal guide policy updates more reliably than the absolute values. DeepSeek-R1's success has already validated GRPO's effectiveness on reasoning tasks, and here it is extended to the visual reasoning domain.

To this end, the paper designed three categories of rewards—format rewards (checking whether the primitive syntax is valid and whether it repeats), quality rewards (LLM judging whether the reasoning is redundant, self-contradictory, or fabricating ground truth to cheat), and accuracy rewards (task-specific, such as smooth exponential rewards for counting and bidirectional trajectory rewards for path tracing to prevent cheating). The key detail is: the RL stage does not directly supervise the intermediate boxes or points, but instead indirectly guides the formation of intermediate reasoning steps through feedback on final behavior quality—this design philosophy of "not directly supervising intermediate steps, only rewarding final behavior quality" is consistent with the core stance of the latter in the debate between Process Reward Models (PRM) and Outcome Reward Models (ORM): PRM gives rewards for each intermediate reasoning step, requiring large amounts of high-quality step-level annotations and being prone to being "gamed" by the model (obtaining rewards by satisfying step form rather than actual reasoning quality); ORM only rewards the correctness of the final answer, with low annotation cost and difficulty being formally deceived, but placing higher demands on the model's intermediate reasoning ability—DeepSeek's choice here shows that as long as the final reward design is precise enough, the model is capable of spontaneously learning reasonable intermediate steps, including when to use bounding boxes and when to use point sequences. Finally, through rejection sampling fine-tuning and on-policy distillation, the two experts are merged into one unified model.

The benchmark results lead by a large margin on tasks where "referencing is the bottleneck":

• PixmoCount counting: 89.2, slightly higher than Gemini 3 Flash;
• Fine-grained counting: 88.7, the best among compared models;
• Spatial reasoning: 98.7;
• Maze navigation: 66.9%, while most frontier models achieve only 49-51%;
• Path tracing: 56.7%, far surpassing GPT (46.5%), Gemini 3 Flash (41.4%), and Claude (30.6%).

The "Scratchpad" Moment for Visual Reasoning

The blogger offers an elegant analogy: this paper's significance for visual reasoning is like the Scratchpad's significance for mathematical reasoning. Except that what the model writes down is not intermediate equations, but intermediate spatial anchors. The Scratchpad method was first proposed by Nye et al. in 2021 and subsequently developed by CoT research. Its core insight is that in complex reasoning tasks, externalizing intermediate states is more important than directly predicting the answer—it allows a limited attention window to handle tasks that exceed its "working memory" capacity. From an information-theoretic perspective, the Scratchpad essentially decomposes reasoning complexity from the "input-output mutual information of a single-step mapping" into a "chain of conditional mutual information across multi-step intermediate variables," where the conditional mutual information of each step is far smaller than the total mutual information of a single step, making it easier to be precisely modeled by a finite-capacity neural network. Visual primitives extend precisely this insight from symbolic space to image space: when the reasoning objects are visual entities rather than numerical or logical relationships, the expressive precision of language tokens is insufficient to serve as reliable intermediate states, whereas bounding box coordinates and point sequences provide precise anchors directly tied to the image coordinate system, compressing the spatial uncertainty in the reasoning chain to near zero.

The reasoning is actually quite simple: if someone guides you through a maze using only pure-language instructions like "slowly go up, then down, then left, then right," with no reference points, you'll quickly lose your way; but once you have coordinates or trackable points, everything becomes easy.

This also reveals a deeper judgment: visual understanding is far from solved, even at the frontier. Models need not only to "see" images but also a precise way to "talk to themselves" about images—and this is exactly what visual primitives provide. The discovery of the reference gap means that the focus of multimodal research is shifting from the perception layer to the reasoning layer, which will have a profound impact on the entire field's architecture design (how to deeply integrate coordinate representations with reasoning chains), data construction (how to generate reasoning trajectories with intermediate visual anchors), and evaluation systems (how to distinguish "seeing" from "reasoning to"). Although the ingredients used in the paper are not new, the way of combining them does indeed bring novelty. As for why DeepSeek retracted the paper after 4 hours, perhaps we'll have to wait for the official version to be released to find out.

Key Takeaways