OmniCapBench: A Deeply Structured Framework for Audio-Visual Caption Evaluation

Multimodal large language models (MLLMs) are rapidly advancing toward continuous audio-visual reasoning, making it an urgent problem to accurately evaluate their capability boundaries. Audio-visual captioning, as an ideal diagnostic task, faces a coupled trade-off dilemma in existing benchmarks: holistic caption scoring covers comprehensively but lacks localization ability, local probing localizes precisely but covers insufficiently, and unconstrained LLM judging introduces instability. OmniCapBench was proposed precisely to break this deadlock.

Core Idea: From Free Text to Atomic Evaluation Units

OmniCapBench (Omni-Video Caption Benchmark) redefines audio-visual caption evaluation as a deeply structured diagnostic framework. Unlike traditional free-text prediction, this benchmark transforms the prediction target into three groups of atomic, verifiable evaluation units: entity references, visual shots, and audio events.

The key to this design is that each evaluation unit can be reliably scored through deterministic constraint checking, combined with localized LLM semantic comparison, ensuring both coverage and precise localization. Compared to previous approaches that rely on holistic scoring or vague LLM judgments, this structured method significantly improves the stability and interpretability of evaluation.

Multimodal large language models (MLLMs) refer to large-scale models capable of simultaneously processing text, images, audio, video, and other modalities as input while generating text output. Representative works include GPT-4o, Gemini, Qwen-VL, etc. Unlike traditional single-modality models, MLLMs need to align semantics of different modalities in a unified representation space. Therefore, their capability boundaries depend not only on the perception accuracy of a single modality but also on the quality of cross-modal fusion and reasoning.

Audio-Visual Captioning requires a model to watch a video while listening to its audio, then generate natural language captions describing the video content. This task inherently involves multiple sub-capabilities such as visual object recognition, audio event detection, temporal localization, and cross-modal association, making it an ideal probe for diagnosing the comprehensive perception ability of MLLMs.

Deterministic constraint checking refers to a scoring method that does not rely on model judgment but is completed by rules or programmatic matching. For example, checking whether a certain annotated entity name appears in the caption, or whether the timestamp of an event falls within the annotated interval. In contrast, LLM semantic comparison allows a certain degree of expression flexibility, where a language model judges whether the generated text and reference annotation are semantically equivalent. OmniCapBench uses a hybrid of both, balancing scoring objectivity with semantic tolerance.

Dataset and Diagnostic Capabilities

OmniCapBench contains 786 densely annotated videos. Although the scale is not massive, the annotation density and degree of structuring enable it to effectively distinguish multiple types of perception errors in MLLMs. Specifically, the benchmark can diagnose the following four typical failure modes:

• Temporal localization failure: The model cannot accurately associate the time point at which an event occurs
• Identity drift: In long videos, the model gradually confuses different entities or speakers
• Cross-modal misalignment: The correspondence between audio and visual information becomes biased
• Hallucinated descriptions: The model generates content that does not exist in the video

This fine-grained error classification capability allows researchers to no longer obtain only a vague "good or bad" score, but to locate specifically which stage the model fails at.

Identity Drift is a typical degradation phenomenon in long video understanding: the model can correctly identify and track an entity (person, object, or speaker) at the beginning of the video, but as time progresses, due to limited context window, attention decay, or insufficient cross-frame association ability, the model gradually confuses that entity with another, or gives incorrect references in subsequent descriptions. Such errors often do not manifest as obvious hallucinations, but rather as hidden, gradual loss of consistency, and are therefore easily averaged out in traditional holistic scoring.

Cross-Modal Misalignment refers to the model incorrectly binding audio information to visual information. For example, A is speaking in the frame while B is the one producing sound in the audio, but the model describes A as speaking; another example is mistaking background music for sound produced by an object in the frame. Such errors directly expose defects in the model's audio-visual alignment mechanism and are one of the core indicators for evaluating omnimodal models.

Performance of Frontier Models Reveals Key Shortcomings

Evaluation results on current frontier MLLMs present a clear pattern: models perform relatively strongly on local perception tasks but are notably weak in long-horizon audio-visual reasoning.

Identity drift and cross-modal misalignment are the two most prominent weak points. This means that as video length increases and involves multi-entity interactions, the model's ability to maintain consistent entity tracking and cross-modal alignment declines significantly. This finding provides clear direction for the development of omnimodal models — future improvements should not focus only on enhancing single-frame perception accuracy, but also need to strengthen long-horizon temporal reasoning and cross-modal consistency maintenance capabilities.

Omnimodal models refer to models capable of uniformly processing text, images, audio, video, and even other perceptual signals within the same architecture, and supporting arbitrary combinations of cross-modal input and output. Compared with multimodal models that support only image-text, omnimodal models need to solve more complex inter-modal temporal alignment and unified representation problems. Current mainstream omnimodal models mostly adopt a concatenated architecture of "visual encoder + audio encoder + large language model backbone," where each modality encoder is independently pretrained and then fused at the language model layer. This architecture performs well on local perception tasks, but when cross-modal long-horizon reasoning is required, due to the lack of explicit temporal alignment supervision, it is prone to degradation in identity tracking and audio-visual binding, which is precisely the core bottleneck revealed by OmniCapBench's diagnostic results.

Implications for Evaluation Methodology

The value of OmniCapBench lies not only in its diagnostic results but also in the shift in its evaluation paradigm. Traditional holistic scoring is convenient but cannot tell developers exactly where the model went wrong; pure local probing is precise but difficult to reflect the model's comprehensive ability.

Breaking down evaluation targets into atomic units and scoring them with a hybrid strategy of deterministic checking plus local LLM semantic comparison may be generalizable to broader multimodal evaluation scenarios. For tasks requiring fine-grained diagnosis of model capabilities, a deeply structured evaluation framework has more practical guiding significance than a single-dimensional score.

One-line Summary

OmniCapBench reconstructs audio-visual caption evaluation with atomic evaluation units to precisely diagnose the long-horizon reasoning shortcomings of MLLMs.

Paragraph Summary

OmniCapBench proposes a deeply structured audio-visual caption evaluation framework that reconstructs traditional free-text scoring into three groups of atomic, verifiable evaluation units: entity references, visual shots, and audio events. It adopts a hybrid scoring strategy combining deterministic constraint checking with local LLM semantic comparison, balancing coverage and localization precision. The benchmark contains 786 densely annotated videos and can diagnose four typical error types: temporal localization failure, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluation of frontier MLLMs shows that models perform relatively strongly on local perception tasks but are notably weak in long-horizon audio-visual reasoning, with identity drift and cross-modal misalignment as the main bottlenecks, pointing the way for improvements in omnimodal models.