Redefining Video Caption Quality Assessment with Multiple-Choice QA: A New VLLM Approach

The Long-Standing Challenge of Video Caption Evaluation

For Visual Large Language Models (VLLMs), objectively evaluating the quality of generated video captions has long been an unresolved problem. Nearly all mainstream evaluation methods follow the same logic: compare the model's generated text against human-annotated ground-truth references, and assign higher scores to closer matches.

This paradigm seems reasonable on the surface, but harbors a fundamental structural flaw. Video descriptions are inherently one-to-many in nature — the same video can be accurately described in countless equally valid ways. Some captions focus on character actions, others on scene context, and still others on object details. All of these can be high-quality descriptions, yet they may be penalized as "incorrect" or "low-scoring" under traditional metrics simply due to different word choices or legitimate shifts in visual focus.

In other words, a genuinely good caption can be systematically undervalued just because it doesn't mirror the exact wording of the reference answer. This lexical mismatch significantly undermines the credibility of existing evaluation frameworks.

The Limitations of One-Dimensional Evaluation

Beyond the misclassifications caused by the one-to-many problem, existing methods suffer from another shortcoming: evaluation dimensions are far too narrow. Most metrics produce a single aggregate score, making it impossible to perform fine-grained analysis of caption quality.

This means that when a caption receives a low score, researchers have little way to pinpoint the actual problem — is information coverage incomplete? Does the description contain hallucinations? Are critical details missing? A single score cannot answer these questions, and therefore cannot guide targeted model improvements. For multimodal model development in fast iteration cycles, an evaluation approach that reveals outcomes but not root causes is clearly insufficient.

Redefining Quality Through Information Fidelity

The core insight proposed by the researchers is to redefine caption quality from the perspective of information fidelity. The key shift here is this: instead of focusing on whether a caption's surface wording matches the reference answer, the evaluation asks whether the caption faithfully and completely conveys the key information present in the video.

Under this definition, a high-quality caption should maximize coverage of the video's content — capturing as many important information points as possible while avoiding the introduction of content that doesn't exist in the video. This perspective shifts the evaluation's center of gravity from "text matching" to "information transfer," more closely aligning with how humans actually judge caption quality.

Multiple-Choice QA: An Elegant Evaluation Vehicle

The most creative aspect of this research is its use of Multiple-Choice Question Answering (MCQA) to operationalize the above evaluation goals.

The underlying logic can be understood as follows: if a caption truly and completely describes a video, then the caption text alone should be sufficient to answer various questions about the video's content. Conversely, if the caption omits key information or contains errors, answering questions based on the caption will produce mistakes. By designing multiple-choice questions around video content and using the caption as the basis for answering, researchers can indirectly and quantitatively measure a caption's information coverage and accuracy.

The elegance of this design lies in how it naturally sidesteps the one-to-many dilemma — regardless of what wording a caption uses or which visual aspect it focuses on, if it can support correct answers to the questions, it has preserved the essential information. At the same time, by designing questions that span different content dimensions, the evaluation naturally gains fine-grained analytical capability.

Implications for Multimodal Research

From a broader perspective, this work addresses a fundamental pain point in multimodal AI evaluation: the tension between the open-ended nature of generative tasks and the requirements of automated assessment. Tasks such as video description, image captioning, and open-ended QA inherently have wide answer spaces — trying to constrain them with rigid reference texts is like forcing a square peg into a round hole.

Using question answering as a proxy task to evaluate generation quality offers a viable alternative path. It transforms "description quality" — which is difficult to quantify directly — into "answer accuracy," which can be objectively determined as correct or incorrect. This preserves the automation and reproducibility of evaluation while better reflecting the true informational value of a caption. For video-understanding VLLMs in rapid development, a fairer and more granular evaluation toolkit is itself foundational infrastructure for driving model progress.

It should be noted that this article is based on a summary of the paper. Specific experimental designs, dataset scales, and comparative results against existing metrics should be obtained from the full paper.