Internalized Visual Thinking (IVT): Deep and Efficient Video Reasoning

The Efficiency Bottleneck of Visual Chain-of-Thought

Multimodal Large Language Models (MLLMs) have made significant progress in recent years on reasoning tasks across spatial, temporal, and embodied environments. MLLMs are unified architectures built on top of large language models (such as GPT-4, LLaMA, etc.), integrating visual encoders (such as ViT, CLIP vision towers) and cross-modal alignment modules to enable the model to simultaneously process text, images, video, and even audio. Representative work includes GPT-4o, LLaVA, Qwen-VL, InternVL, and others, which typically adopt a three-stage architecture of "visual encoder + projection layer + large language model," where the projection layer maps visual features into the language model's embedding space to achieve cross-modal semantic alignment.

Among the many capabilities of these models, one key technique is Visual Chain-of-Thought (Visual CoT). The concept of Chain-of-Thought (CoT) was first proposed by Jason Wei and colleagues at Google Brain in 2022. The core idea is to include step-by-step reasoning examples in prompts, guiding large language models to decompose complex problems into intermediate steps, thereby significantly improving accuracy on tasks like mathematical reasoning and logical reasoning. The community has since developed variants such as Zero-shot CoT (triggered by "Let's think step by step"), Self-Consistency (multi-path voting), and Tree-of-Thought (tree-based search). Visual CoT extends this concept to the multimodal domain—the model not only generates intermediate text steps but also generates intermediate reasoning images, such as annotating key regions, predicting future frames, or drawing spatial relationship diagrams to aid final judgment. This provides the model with an intuitive "visual foresight" mechanism, allowing it to sketch out its understanding of a scene in its "mind's eye" before reaching a final conclusion.

However, this capability comes at a significant cost. Generating intermediate reasoning images means repeatedly invoking image generation modules during inference, resulting in enormous inference overhead. Image generation in Visual CoT typically relies on diffusion models (such as Stable Diffusion, SDXL) or autoregressive image generators (such as discrete visual token generation in Chameleon). Taking diffusion models as an example, generating a 512×512 resolution image typically requires 20-50 denoising iterations, with single-image generation latency of approximately 2-8 seconds on consumer-grade GPUs. If Visual CoT needs to generate 3-5 intermediate images during one inference pass, cumulative latency can reach 10-40 seconds. Even using distilled acceleration models (such as LCM, SDXL-Turbo), latency remains far beyond the scale of pure text reasoning.

For scenarios requiring real-time response, especially proactive video reasoning tasks, this latency is almost unacceptable. Video streams inherently carry strong temporal sensitivity—taking a 30fps video as an example, 10 seconds of latency means the model has already "missed" 300 frames of information—any additional computational burden directly impacts system usability.

This raises a core question: Can a model learn to "think visually" during training, while directly providing conclusions during inference without actually generating intermediate images?

The Core Design Philosophy of IVT

To answer this question, researchers proposed the Internalized Visual Thinking (IVT) framework. This is a post-training solution. Post-training refers to techniques that inject specific capabilities or align with human preferences through additional training stages after a model has completed large-scale pretraining. Common post-training methods include: supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), direct preference optimization (DPO), and various knowledge distillation schemes. The advantage of post-training is that it doesn't require training models from scratch, but rather "stacks" new capabilities on top of existing powerful foundation models, with relatively controllable training costs. As a post-training framework, IVT can be applied to any pretrained model that already possesses basic multimodal capabilities, offering good generalizability and scalability.

The core philosophy of IVT can be summarized as: Internalize the capability of visual thinking into model parameters, rather than explicitly externalizing it as images during inference.

Think During Training, Direct Output During Inference

Traditional Visual CoT operates in a "generate images while reasoning" mode, where images are both the product and carrier of thinking. IVT decouples these two stages: during training, the model learns to perform internal reasoning like Visual CoT through joint optimization of text prediction and visual thinking objectives; but during inference, it can directly output text conclusions, skipping the expensive image generation step.

The elegance of this design lies in preserving the reasoning quality improvements brought by Visual CoT while eliminating the largest performance bottleneck. The model uses the scaffold of visual foresight during "learning," but travels light during "application."

This approach has deep conceptual ties to Knowledge Distillation. Knowledge distillation, proposed by Hinton and colleagues in 2015, centers on "compressing" knowledge from a large model (teacher model) into a small model (student model). IVT can be viewed as a "self-distillation" variant—the model plays the "teacher" role during training (leveraging explicit visual intermediate products for rich reasoning) and the "student" role during inference (relying only on internalized parameters for direct output). This thinking also resonates with recent discussions about the tradeoff between "inference-time compute" and "training-time compute": OpenAI's o1 model takes the route of increasing inference-time computation, while IVT takes the route of front-loading computation into the training phase—representing two distinctly different directions for optimizing inference performance.

How Joint Optimization Works

As a post-training framework, IVT's key lies in "joint optimization." The model no longer treats visual reasoning and text generation as two independent tasks, but trains them collaboratively under a unified optimization objective. The signal of visual thinking is encoded into the model's internal representations, allowing text prediction to implicitly "benefit" from visual-level understanding—even though these visual intermediate products won't actually be rendered during final inference.

Why IVT Is Crucial for Video Reasoning Scenarios

Proactive video reasoning is a particularly demanding application scenario. Unlike passively answering questions about videos, proactive reasoning requires models to continuously track video streams, anticipate event trajectories, and proactively respond at appropriate moments. Traditional video understanding tasks (such as VideoQA, video captioning) belong to the passive paradigm: given a complete video and a question, the model answers retrospectively. Proactive video reasoning requires models to process video in a streaming manner, making predictions and interventions before events fully unfold. Typical scenarios include: hazard warnings in autonomous driving (requiring judgment hundreds of milliseconds before collision), real-time decision-making for surgical assistance robots, and abnormal behavior detection in intelligent security. The core challenge of such tasks is that models must perform "online inference" with incomplete information, with strict constraints on end-to-end latency—typically requiring responses within 100-500 milliseconds.

Imagine an embodied agent navigating a dynamic environment, or a surveillance system needing to alert at the instant an anomaly occurs. Embodied agents are AI systems with perception and action capabilities in physical or simulated environments, including service robots, drones, and autonomous vehicles. Unlike pure software AI, embodied agents face closed-loop "perception-decision-execution" challenges: the environment continuously changes, decisions must be made in real-time, and incorrect decisions may lead to irreversible physical consequences. Current mainstream embodied AI research (such as Google's RT-2, Figure 01 humanoid robot) is exploring how to efficiently deploy large model reasoning capabilities into robot control loops, and IVT's solution to "reducing latency while preserving reasoning depth" precisely addresses one of the key bottlenecks for scaling embodied AI deployment.

If each reasoning step requires first generating an intermediate image, accumulated latency would make the system "slow to react," losing the meaning of proactivity. IVT addresses this pain point by internalizing visual thinking: achieving both the depth of visual foresight and the speed of direct reasoning.

Technical Significance and Industry Impact of IVT

From a broader perspective, IVT represents an important paradigm shift in multimodal reasoning research. Over the past few years, the community has tended to make model reasoning processes "explicit"—whether text CoT or Visual CoT, both emphasize externalizing the thinking process, both for interpretability and for perceived performance gains.

But there has always been tension between explicit reasoning and deployment efficiency. IVT's proposed "externalize during training, internalize during inference" approach provides an elegant compromise for this contradiction. It prompts us to consider: The acquisition of reasoning capability and the presentation of reasoning process can perhaps be separated. Models can fully leverage rich intermediate supervision signals during training while running in the most streamlined way during actual service.

Implications for Practical Deployment

For engineering teams focused on deployment, the value of methods like IVT is very direct. It means significantly reducing inference costs and latency for video AI applications without sacrificing reasoning quality, which is extremely attractive for edge devices, real-time systems, and large-scale service scenarios.

In actual engineering deployment, inference cost is a core factor determining AI product feasibility. Cloud GPU inference costs approximately $0.5-4 per hour (depending on GPU model), while edge devices (such as NVIDIA Jetson series, mobile NPUs) have even more limited computing power, typically only supporting INT8/INT4 quantized model execution. Visual CoT requires additionally running image generation models, which is almost infeasible on edge devices—even a lightweight diffusion model requires several GB of VRAM and several seconds of inference time. By eliminating the need for image generation during inference, IVT makes model deployment on edge devices practically feasible, which holds important significance for scenarios like smart cameras, AR glasses, and in-vehicle computing platforms.

Of course, as a post-training technique, IVT's actual effectiveness depends on training data quality, visual thinking objective design, and controllability of internalization degree. These details will determine whether it can stably reproduce the reasoning gains of explicit Visual CoT in real tasks.

Summary

Internalized Visual Thinking (IVT) provides a new path for multimodal large model video reasoning that balances quality and efficiency. By jointly optimizing text prediction and visual thinking during training, it enables models to "learn" visual reasoning while directly outputting results during inference, fundamentally circumventing the high inference overhead of Visual CoT. For latency-sensitive proactive video reasoning, this approach is particularly fitting. As multimodal applications advance toward real-time and embodied directions, achieving balance between "deep thinking" and "rapid response" will remain an ongoing research focus, and IVT represents a valuable exploration in this direction.