OmniCapBench
An audio-visual captioning evaluation benchmark that structures caption evaluation into three categories of atomic evaluation units: entity reference, visual shot and audio event. It combines deterministic constraint checking with local LLM semantic comparison, enabling fine-grained diagnosis that covers both coverage and grounding ability.
Timeline (last 90 days)
OmniCapBench (Omni-Video Caption Benchmark) is a deeply structured diagnostic framework for audio-visual caption evaluation, whose prediction targets are three groups of atomic, verifiable evaluation units: entity reference, visual shot, and audio event.
OmniCapBench contains 786 densely annotated videos.
OmniCapBench adopts a hybrid scoring strategy combining deterministic constraint checks with localized LLM semantic comparison, in order to balance coverage and precise localization.
OmniCapBench can diagnose four typical failure modes: temporal localization failure, identity drift, cross-modal misalignment, and hallucinated description.
Evaluation results on current frontier MLLMs show that the models perform strongly on local perception tasks but are clearly weak in long-horizon audio-visual reasoning.
The idea of decomposing evaluation targets into atomic units and scoring with a hybrid strategy of deterministic checks plus localized LLM semantic comparison can be extended to broader multimodal evaluation scenarios.
All Facts (6)
OmniCapBench (Omni-Video Caption Benchmark) is a deeply structured diagnostic framework for audio-visual caption evaluation, whose prediction targets are three groups of atomic, verifiable evaluation units: entity reference, visual shot, and audio event.
50%UnverifiedOmniCapBench contains 786 densely annotated videos.
50%UnverifiedOmniCapBench adopts a hybrid scoring strategy combining deterministic constraint checks with localized LLM semantic comparison, in order to balance coverage and precise localization.
50%UnverifiedOmniCapBench can diagnose four typical failure modes: temporal localization failure, identity drift, cross-modal misalignment, and hallucinated description.
50%UnverifiedEvaluation results on current frontier MLLMs show that the models perform strongly on local perception tasks but are clearly weak in long-horizon audio-visual reasoning.
50%UnverifiedThe idea of decomposing evaluation targets into atomic units and scoring with a hybrid strategy of deterministic checks plus localized LLM semantic comparison can be extended to broader multimodal evaluation scenarios.
50%