OmniCompass: How an Omni-Modal Video Understanding Benchmark Evaluates 50 MLLMs

The ability of Multimodal Large Language Models (MLLMs) to understand video is evolving rapidly, but there has long been a lack of a sufficiently complete ruler to systematically measure what they have truly "understood." A recently released study proposes OmniCompass—a holistic benchmark for omni-modal video grounding and understanding, aiming to fill the gaps in existing evaluations regarding audio, spatiotemporal grounding, and cross-modal consistency.

Why Existing Video Understanding Benchmarks Fall Short

Most current mainstream video understanding evaluations focus on a single dimension: some lean toward video question answering, some only examine temporal grounding, and others rely entirely on visual signals while ignoring audio. This fragmentation makes it difficult for researchers to determine whether a model has actually "seen the picture," "heard the sound," or merely guessed the right answer at the language level.

The core premise of OmniCompass is that omni-modal video understanding must simultaneously answer three questions—when the target appears, where it is, and what is happening. Only when an evaluation covers these three complementary dimensions while also examining the joint use of visual and auditory evidence can it provide a systematic diagnosis of a model.

Omni-modal refers to a model's ability to simultaneously receive and jointly process multiple modalities of input signals—text, images, video, audio, and more—rather than relying on just one or a few of them. The closely related concept is "multimodal," but omni-modal typically emphasizes a broader coverage of modalities, particularly requiring that audio and visual signals be genuinely fused within the same reasoning process rather than simply concatenated as features.

The technical challenge of omni-modal understanding lies in cross-modal alignment: the model needs to establish semantic correspondences between audio events (such as glass shattering or a door closing) and visual content, and then complete grounding or question answering on that basis. Currently, most MLLMs' audio encoders and visual encoders are mutually independent during pretraining, causing joint reasoning to easily degrade into "trusting one modality by default." This is precisely why OmniCompass distinguishes between VL and AV tracks and separately examines audio-visual tasks.

Two Evaluation Tracks and Three Task Types

OmniCompass designs two parallel evaluation tracks to distinguish whether a model possesses audio processing capabilities.

OmniCompass-VL targets vision-language models, where all evidence required for tasks comes from visual information. OmniCompass-AV targets omni-modal models, requiring models to jointly use audio and visual evidence to answer.

Each track contains three types of complementary tasks:

• Temporal Grounding (TG): determining when a referenced target appears in the video
• Spatial Grounding (SG): determining the spatial location of a target within the frame
• Question Answering (QA): understanding the content of the event itself

The QA portion is further subdivided. In OmniCompass-VL, question answering covers 5 perceptual categories and 3 reasoning categories; in OmniCompass-AV, it covers 3 perceptual categories and 2 reasoning categories. This classification allows the evaluation to distinguish between two different levels of capability: "being able to see/hear" versus "being able to reason clearly."

Temporal Grounding (TG) requires the model to output the time interval during which a referenced target appears in the video, typically expressed as start and end timestamps, with evaluation metrics commonly using IoU (Intersection over Union) or mIoU. It tests whether the model can anchor a language description to the correct segment on a continuous timeline, rather than merely judging "whether an object exists in the video."

Spatial Grounding (SG) requires the model to give the position of a target within the frame, typically represented as a bounding box, with evaluation metrics commonly using IoU or accuracy. It complements temporal grounding: the former answers "when," while the latter answers "where." Both impose high demands on a model's fine-grained perception capabilities, and both are among the weakest areas of current MLLM performance.

Question Answering (QA) focuses on the content of the event itself, belonging to relatively coarse-grained semantic understanding, further subdivided into perceptual categories (identifying objects, attributes, actions) and reasoning categories (causality, intent, temporal logic), thereby measuring "being able to see/hear" and "being able to reason clearly" as separate capabilities.

Data Composition and Paired Evaluation Design

In terms of data scale, OmniCompass-VL contains 820 videos and 2,460 instances, covering multiple domains and three duration types: short, medium, and long. Among them, the subset rich in narrative and audio information constitutes OmniCompass-AV, totaling 240 videos and 720 instances.

One design worth noting is "paired evaluation." Each instance is evaluated under two settings: text-only and text+image. Both point to the same target but designate it through textual description and visual-language guidance respectively. In this way, whether a model truly understands the target's reference—rather than relying on a particular prompt form as a shortcut—can be directly tested. All instances undergo human annotation and multiple rounds of quality verification.

What the Evaluation of 50 MLLMs Reveals

The researchers evaluated 50 MLLMs, and the results present several clear conclusions.

Open-source models overall still lag behind closed-source models, and this gap is particularly pronounced on audio-visual understanding tasks. The introduction of audio information is not merely adding an input channel—it imposes higher demands on a model's cross-modal alignment capabilities, which is precisely the shortcoming of current open-source models.

Precise temporal and spatial grounding remains difficult. Even with leading models, accurately pointing out "when" a target appears and "where" it is remains an unresolved core challenge. This shows that models perform passably on coarse-grained semantic understanding, but still have clear gaps on tasks requiring precise localization.

Predictions are inconsistent under paired settings. When the same target is described in plain text or guided by text+image, the model's predictions become inconsistent. This exposes a deeper problem: the model's understanding of targets is not stable and may depend heavily on prompt form rather than genuine multimodal semantics.

Paired evaluation is a controlled-variable experimental design: the same target instance is presented to the model in two forms—plain text description and text+image—and if the model gives contradictory predictions under the two settings, it indicates that its understanding of the target reference is unstable and may depend on the surface form of the prompt rather than genuine multimodal semantic understanding.

This approach draws on the tradition of "robustness probes" in natural language processing—testing whether a model has learned genuine concept representations by changing the expression of the input without changing its semantics. In video understanding scenarios, the two paired prompts impose language guidance and visual-language guidance respectively, so inconsistent predictions often point to deficiencies in cross-modal alignment or reference resolution rather than mere knowledge gaps. This design makes the evaluation conclusions more diagnostically valuable.

What This Benchmark Means

The value of OmniCompass lies not in providing a new leaderboard, but in decomposing "omni-modal video understanding" into diagnosable dimensions. Whether audio is genuinely used, whether spatiotemporal grounding is precise, whether cross-modal reference is stable—these questions are examined side by side within the same framework.

For model developers, this design provides more fine-grained failure localization: when a model answers incorrectly, it becomes possible to determine whether the problem lies in auditory fusion, spatial perception, or prompt-form dependence. For evaluation researchers, the paired evaluation approach also offers a reusable paradigm for testing model robustness. Omni-modal understanding is far from solved, but at least there is now a more complete coordinate system.

One-line Summary

OmniCompass proposes an omni-modal video understanding benchmark, using dual tracks, three tasks, and paired evaluation to systematically diagnose 50 MLLMs.

Paragraph Summary

OmniCompass is a holistic benchmark for omni-modal video grounding and understanding, aiming to fill gaps in existing evaluations regarding audio, spatiotemporal grounding, and cross-modal consistency. It designs two tracks, OmniCompass-VL and OmniCompass-AV, respectively examining vision-language models and omni-modal models, with each track containing three complementary tasks: temporal grounding, spatial grounding, and question answering. In terms of data scale, VL contains 820 videos and 2,460 instances, while the AV subset contains 240 videos and 720 instances, and it uses paired text-only and text+image settings to test whether models depend on prompt form. Evaluation of 50 MLLMs shows: open-source models overall lag behind closed-source models, with an especially large gap in audio-visual understanding; precise spatiotemporal grounding remains difficult even for leading models; and predictions are inconsistent under paired settings, exposing instability in cross-modal reference understanding.