VICE: Concept-Level Video Similarity Estimation via Video-Language Entailment

Video similarity estimation is a fundamental problem in multimedia understanding, but current Large Multimodal Models (LMMs) are mostly trained for single-video tasks and often struggle when directly comparing two videos. A recent paper proposes VICE (Video Caption Entailment for video similarity estimation), which uses a dual-branch design to transform cross-video comparison into a video-language entailment task, surpassing models that process two videos simultaneously on concept-level similarity.

Why video similarity estimation needs new approaches

Traditional methods mostly focus on global similarity—whether two videos are similar overall. In practice, however, users care more about similarity along specific concept dimensions, such as whether two videos share the same "activity" or the same "location." This concept-level similarity is more challenging because it requires the model to parse video content at a fine-grained level rather than simply comparing overall features.

Existing LMMs are primarily trained with single-video objectives and lack cross-video reasoning ability. Although prior research has introduced the concept-level similarity problem, no work has specifically leveraged LMMs to explicitly solve it. VICE aims to fill this gap.

Large Multimodal Models (LMMs) refer to large-scale pretrained models that possess both visual and language understanding capabilities, with typical examples including GPT-4V, LLaVA, Qwen-VL, and others. These models are usually trained on image-text or video-text pairs and can perform tasks such as image captioning, visual question answering, and video description generation. However, the training objectives of mainstream LMMs almost all revolve around a single visual input—input one video or one image, output the corresponding text. The model rarely sees samples during training where "two visual samples are input simultaneously and compared," so its cross-sample reasoning ability has not been explicitly optimized. This is precisely the technical background behind the VICE paper's emphasis that "LMMs have strong native single-video understanding but weak cross-video comparison ability." Understanding this is essential to appreciate why decomposing dual-video comparison into two single-video-text judgments is a design that leverages strengths and avoids weaknesses.

VICE's core design: dual-branch entailment reasoning

VICE adopts a dual-branch architecture, where each branch processes one video and the text description (caption) of the other video. Specifically, one branch receives video A and the caption of video B, while the other branch receives video B and the caption of video A. Each branch uses video-language entailment to judge whether the two are aligned on a specific concept, outputting an alignment measure.

The results from the two branches are then fused to generate the final similarity score, and the model also provides a textual reasoning explanation. This design fully leverages the LMM's native single-video understanding capability while achieving fine-grained reasoning through cross-comparison between video and text.

The key insight is: rather than having the model view two videos simultaneously (which is difficult for current LMMs), it is better to have the model view one video at a time and judge whether it "entails" the description of the other video. Entailment naturally suits measuring the degree of concept alignment.

Video-language entailment is borrowed from the textual entailment task in natural language processing. In NLI (Natural Language Inference), the system needs to determine whether a given premise and hypothesis are in an entailment, contradiction, or neutral relationship. When transferred to the video-language setting, the premise becomes a video and the hypothesis becomes the text description of another video, and the model needs to judge whether the video content "supports" the description holding on a specific concept dimension. Entailment naturally has directional and fine-grained characteristics: it does not require overall similarity but instead examines whether specific assertions in the description are satisfied by video evidence. This makes it more suitable for concept-level alignment judgment than global measures like cosine similarity, and also provides a theoretical basis for dual-branch cross-validation (A entails B's description, B entails A's description).

Experimental performance: single-video reasoning outperforms simultaneous dual-video input

The paper conducted experiments on two benchmarks, ConViS-Bench and StepDiff. Results show that VICE outperforms models that process two videos simultaneously. This finding is noteworthy: it indicates that for concept-level similarity estimation, a strategy of split processing followed by fusion is more effective than direct joint encoding.

This also indirectly shows that current LMMs still have limitations in cross-video reasoning, and through an indirect approach like video-language entailment, it is possible to circumvent the model's structural shortcomings and leverage its strengths in single-video understanding.

ConViS-Bench is a benchmark specifically designed to evaluate concept-level video similarity, covering multiple concept dimensions such as activity, location, and objects, requiring the model to judge whether two videos are consistent on a specified concept. StepDiff focuses on fine-grained perception and description of video differences. The common feature of these two benchmarks is that they not only examine whether the similarity scores given by the model are accurate but also whether the model can identify "on which concept they are similar or different." VICE's advantage on these two benchmarks demonstrates that the split dual-branch design has practical gains in scenarios requiring fine-grained concept judgment, rather than being merely theoretically elegant.

Implications for multimodal research

VICE's approach provides a reusable paradigm: when a model is not good at directly comparing two complex inputs, the comparison can be transformed into a single-input task that the model is better at, plus a relational judgment. Video-language entailment serves as the bridge for relational judgment here.

Additionally, the model's ability to output textual explanations enhances the interpretability of results, which has practical value for application scenarios such as video retrieval and recommendation systems. In the future, this dual-branch entailment design may be extended to more cross-modal comparison tasks, such as image-video similarity or audio-video alignment.

The paper's code and further details have not yet been released, but this direction is worth watching.

One-line Summary

VICE uses dual-branch video-language entailment to transform cross-video comparison into single-video reasoning, improving concept-level similarity estimation.

Paragraph Summary

To address the lack of cross-video comparison ability in Large Multimodal Models (LMMs), VICE proposes transforming video similarity estimation into a video-language entailment task. Its dual-branch architecture has the model view video A and judge whether it entails the description of video B, and view video B and judge whether it entails the description of video A, then fuses the two alignment measures to obtain a similarity score and outputs a textual reasoning explanation. This design avoids the difficulty of LMMs encoding two videos simultaneously and fully leverages their single-video understanding strengths. On the ConViS-Bench and StepDiff benchmarks, VICE outperforms models that process two videos simultaneously, indicating that the split-then-fuse strategy is more effective for concept-level similarity estimation and provides a reusable paradigm for cross-modal comparison tasks.