Why Is Google the Only One Excelling at Video Understanding LLMs? The Triple Barrier of Data, Architecture, and Cost

An Overlooked AI Capability Gap

In the race for Large Multimodal Models (LMMs), the ability to understand text, images, and even audio has become table stakes for all major players. Large Multimodal Models are AI systems that can simultaneously process and understand multiple forms of information (modalities)—traditional Large Language Models (LLMs) were initially limited to text, while multimodal models extend their capabilities to images, audio, video, and more. There are two main technical approaches to achieving multimodality: one is a "late fusion" architecture that connects visual encoders to a pre-trained text model via connectors; the other is a "natively multimodal" architecture that ingests multiple modalities from the very beginning of training. The performance difference between these two approaches on video understanding is one of the core topics of this article.

However, when the conversation turns to video input understanding and reasoning, a clear divergence emerges. A Reddit user posed a highly representative question:

"Why does it seem like only Google has built a decent video understanding/reasoning model? Anthropic, OpenAI, etc. (not sure about Chinese companies) don't seem to have released good video models. Is it a compute issue? An ROI issue? Or a data availability issue?"

This question touches on a critical blind spot in current multimodal AI development. Indeed, Google's Gemini series has established a significant lead in native video understanding, and the reasons behind this are far more complex than simply "who works harder."

Where Does Google's Video Understanding Advantage Come From?

The Unparalleled YouTube Data Asset

Any discussion of Google's lead in video cannot avoid one key factor: YouTube. As the world's largest video platform, YouTube sees over 500 hours of video uploaded every minute, with cumulative video content numbering in the billions. It provides Google with an almost unrivaled source of video training data—not just massive amounts of video content, but crucially the accompanying subtitles, titles, descriptions, tags, and even user comments, forming a natural "video-text" paired dataset.

Multimodal model training is highly dependent on cross-modal alignment data. Cross-modal alignment is a core concept in multimodal learning, referring to the process of teaching a model to map semantically equivalent information from different modalities into a shared representation space. The CLIP model achieved image-text alignment through contrastive learning on 400 million image-text pairs, becoming the foundation for subsequent multimodal models. But video-text alignment faces greater challenges: the temporal nature of video means alignment must occur not only at the spatial level but also precisely match at the temporal level—which segment of text corresponds to which time period of the video. This requires far more fine-grained annotation data than image-text pairing.

The image domain has abundant image-text pairs (such as images with alt text on web pages), but high-quality "video-text" paired data is extremely scarce on the open internet. YouTube's Automatic Speech Recognition (ASR) system can generate timestamp-aligned subtitles for the vast majority of videos, essentially providing fine-grained temporal alignment of "video frames to speech text." Additionally, YouTube has content classification systems, video chapter markers, and user interaction data (likes, comments, watch duration) that can all serve as weak supervision signals for training. Google DeepMind's previous research (such as Flamingo, VideoPoet, etc.) extensively used YouTube data—this data accumulation is the result of long-term strategic investment. This is a structural advantage that OpenAI and Anthropic cannot replicate in the short term.

The Native Multimodal Architecture Design Choice

Gemini was positioned from the start as a "natively multimodal" model, meaning it simultaneously ingests text, images, audio, and video during the pre-training phase, rather than bolting on visual encoders after the fact to a text model. This architectural difference gives Gemini better native understanding of temporally information-dense inputs like video.

Specifically, late fusion architectures (such as LLaVA, early versions of GPT-4V) work by first using a pre-trained visual encoder to convert images into a series of visual tokens, then mapping them to the language model's embedding space through a lightweight alignment module, and finally having the language model process everything uniformly. The advantage of this approach is that it can reuse mature text and vision models with lower training costs; the disadvantage is that visual-language fusion occurs at relatively shallow layers, making it difficult for the model to learn deep cross-modal interaction patterns. Native multimodal architectures allow information from different modalities to attend to each other at every Transformer layer, enabling the model to learn inter-modal correlations from the ground up—this is particularly important for complex video scenarios where visuals, sound, and text appear simultaneously.

The key difference between video and static images lies in the temporal dimension—understanding video requires the model to capture motion, causality, and event progression between frames. This places extremely high demands on the model's context length and temporal modeling capabilities. Gemini's sustained investment in long context windows (million-level tokens) provides exactly the infrastructure needed for processing long videos. Gemini 1.5 Pro supports up to 1 million tokens of context window, with subsequent versions even exploring 2 million tokens. Key technologies enabling ultra-long context include: Ring Attention (distributing long sequences across multiple devices for chunked attention computation), Mixture-of-Experts (MoE) sparse activation architectures (reducing the actual parameters activated per token), and improved positional encoding schemes (such as optimized extrapolation capabilities of RoPE). These combined technologies enable Gemini to "read in" hours of video content at once and perform global reasoning, rather than processing only short clips.

Why Are OpenAI, Anthropic, and Others Temporarily Behind?

The Reality of Compute and Cost Constraints

Video is the most information-dense and most expensive modality to process. One minute of video, sampled at several frames per second, can be equivalent to processing hundreds of images. The compute overhead for training and running inference on video models far exceeds that of pure text or image models.

Video tokenization is the process of converting continuous video streams into discrete token sequences for Transformer architectures to process. Mainstream methods include: uniformly sampling keyframes and encoding them with ViT, using 3D convolutional networks to extract spatiotemporal features, or using video VQ-VAE for compression. The core contradiction is: the higher the sampling rate, the more complete the temporal information preserved, but the number of tokens generated grows linearly or even quadratically. A 10-minute video sampled at 2 frames per second with 256 tokens per frame would produce over 300,000 tokens—far exceeding most models' context window limits, necessitating efficient compression strategies such as temporal pooling, dynamic frame selection, or hierarchical encoding.

For companies like OpenAI and Anthropic, with limited compute resources, it's rational to prioritize investment in scenarios with the strongest user demand and most direct commercial returns (such as code generation, text conversation, image understanding). While video understanding has broad prospects, the current ROI remains unclear, making it unlikely to become a core competitive battleground in the near term.

The Structural Difference in Video Data Barriers

As mentioned above, the barrier to acquiring video-text paired data is extremely high. Companies that don't control large video platforms need to build training sets through purchasing, scraping, or synthesis—approaches that are difficult to match YouTube's natural data in terms of cost and quality. This constitutes a nearly insurmountable data moat.

You may not have noticed that OpenAI actually excels in video generation (Sora), which shows they don't lack technical expertise in the video domain, but rather haven't yet invested equivalent resources in video understanding and reasoning. Video generation (such as Sora, Runway Gen-3) is primarily based on Diffusion Models or autoregressive generation architectures, with the core task being synthesizing visually realistic, temporally coherent video frame sequences from text descriptions—during training, the model learns pixel-level distributions and motion patterns of video. Video understanding, on the other hand, requires the model to extract semantic information from video, perform logical reasoning and knowledge integration—for example, answering causal reasoning questions like "why did the person in the video suddenly turn around." The former is evaluated on aesthetic quality and physical consistency, while the latter is evaluated on semantic accuracy and reasoning depth. Although both involve video data, the required model capabilities, training objectives, and evaluation systems are fundamentally different—OpenAI's spatiotemporal modeling capabilities demonstrated in Sora cannot be directly transferred to video understanding tasks.

Core Technical Challenges of Video Understanding LLMs

The Complexity of Cross-Frame Temporal Reasoning

The core challenge of video understanding lies not in "understanding each frame" but in cross-frame reasoning. For example, judging the intent behind an action, understanding the context of a conversation, tracking an object's trajectory—all require the model to integrate and reason across the temporal dimension. This is far more difficult than understanding a single static image.

From a technical perspective, cross-frame reasoning requires a combination of capabilities: temporal causal modeling (Event A causes Event B), object permanence tracking (continuous identification even when the target is briefly occluded), action semantic understanding (distinguishing temporally opposite actions like "picking up" and "putting down"), and long-range dependency capture (clues at the beginning of a video may only reveal their significance at the end). The current mainstream self-attention mechanism has O(n²) computational complexity when processing ultra-long sequences, making direct global attention over all frames impractical—researchers are therefore exploring compromises such as hierarchical attention and sparse attention.

The Challenge of Ultra-Long Context Windows

Processing a complete video often means requiring extremely long context windows. How to compress video information while maintaining reasoning quality, and how to efficiently tokenize video frames, remain engineering and algorithmic challenges that haven't been fully solved. Google's lead in long-context technology indirectly supports the establishment of its video capabilities.

Current solutions mainly follow three directions: first, "compress then reason," using video summarization modules to compress long videos into a few keyframes or semantic tokens before feeding them into the LLM; second, "streaming processing," splitting videos into short segments for sequential processing while maintaining contextual coherence through memory mechanisms; third, "end-to-end long context," directly expanding the model's context window to accommodate complete videos. Google's Gemini primarily takes the third approach, supporting million-level token processing through synergistic innovation in hardware (TPU clusters) and algorithms (Ring Attention, MoE).

How Will the Video Understanding Competitive Landscape Evolve?

From Reddit community discussions, it's clear that the industry's attention to video understanding is rising. While Google currently holds the leading position, this lead stems more from data assets and architectural choices as first-mover advantages, rather than an insurmountable technological chasm.

As compute costs decline, video data acquisition methods diversify (such as synthetic data, multi-party data partnerships), and the commercial value of video understanding applications (video search, content moderation, education, security, etc.) becomes clearer, other players will very likely accelerate their efforts. Chinese companies' data accumulation in the short-video ecosystem could also become a significant force in the future. China has the world's most active short-video ecosystem—Douyin (TikTok's Chinese version) has over 600 million daily active users, and platforms like Kuaishou and Bilibili have also accumulated massive video data. ByteDance, Alibaba, Tencent, and other companies have deep technical expertise in video recommendation algorithms, and their recommendation systems inherently rely on semantic understanding of video content. Since 2024, multiple Chinese AI companies have released multimodal models with video understanding capabilities, such as Qwen-VL series, InternVL, and Step-1V. Additionally, China's extensive deployment of video surveillance and security applications has generated substantial real-world demand for video analysis, providing clear commercialization scenarios and deployment motivation for video understanding models.

It's foreseeable that video understanding will become the next frontier in multimodal LLM competition. Google's lead is a temporary moat, not a permanent barrier.