PhyMetric: Diagnosing Text-to-Video Physical Plausibility with Scene-Level QA

Text-to-video (T2V) generation models can already produce visually convincing clips, but visual realism does not equal physical correctness. Given a prompt like "a ball rolls off the table," a model might show the ball hovering, bouncing backward, or vanishing into thin air—failures that violate physical causality but are hard to pinpoint with traditional holistic quality scores. PhyMetric proposes a diagnostic benchmark that reframes physical plausibility evaluation as verifiable scene-level question answering, making "which physical law was violated and why the model failed" traceable.

From Holistic Scoring to Scene-Level QA

The main problem with existing T2V evaluation is its "black box" nature: whether a human gives a 1–5 plausibility rating or a vision-language model (VLM) makes a broad quality judgment, the result is just a scalar. That score can't specify which type of physical behavior was tested, nor can it explain the specific step where the model failed.

PhyMetric's approach is to break each generated video down into answerable, specific questions. Each scene is paired with eight binary (yes/no) questions covering different dimensions of physical behavior such as gravity, collision, deformation, fluids, and lighting. During evaluation, a VLM serves as the judge, answering on question-aware sampled frames, and accuracy is then aggregated across four levels: scene, dimension, physical category, and model. This design replaces vague questions like "is this video physically plausible?" with decidable propositions like "did the ball change direction after contacting the table?"

A vision-language model (VLM) is a multimodal model that accepts both image/video and text inputs and outputs text responses; typical examples include GPT-4V, LLaVA, and Qwen-VL. Unlike models that handle a single modality, VLMs are pretrained on large-scale image-text pairs to align visual content with language semantics, giving them the ability to "look at an image and answer questions." In PhyMetric, the VLM plays the role of an automatic judge: given sampled frames and a binary question, it outputs "yes" or "no." The reliability of this role directly determines the evaluation results, which is why the paper specifically uses ablation experiments to examine how question phrasing affects VLM judgment. Understanding the boundaries of VLM capabilities (for instance, its weakness in fine-grained temporal reasoning) is a prerequisite for understanding why scene-level questioning is necessary and why the gains from physical priors fluctuate with judge capability.

Benchmark Scale and Structure

PhyMetric contains 1,115 scenes spanning 6 domains and 50 physical categories, paired with 8,920 answer-balanced binary questions distributed across 12 evaluation dimensions. Answer balance means the yes/no labels for each question are paired and controlled, preventing models from scoring high just by guessing "yes"—this is crucial to the benchmark's credibility.

Beyond the questions themselves, each scene also comes with a structured physical prior that explicitly states four pieces of information: the physical laws governing the scene, the expected temporal sequence, observable feature signals, and common failure modes. This prior serves both as a reference before evaluation and as a way to explain afterward why the model answered incorrectly.

Answer balance is a common quality-control technique when constructing multiple-choice or true/false benchmarks. If a question set is dominated by affirmative answers like "yes," a model can achieve high accuracy just by always answering "yes," and the benchmark loses its discriminative power. Answer balance requires that the yes/no labels for each type of question be roughly paired in distribution, so that the expected accuracy of random guessing stays around 50%, and a model must genuinely make judgments to exceed this baseline. This is also why the paper treats "VLM judge accuracy dropping to random guessing level" as strong evidence—it means that under generalized questioning, the judge has almost no effective discriminative ability. Similar balancing strategies are widely adopted in benchmarks such as visual question answering and commonsense reasoning, and serve as one of the fundamental safeguards for the credibility of evaluation protocols.

Binary QA Is Not Optional

The paper's most striking finding comes from an ablation-style controlled experiment. The researchers replaced each targeted question with a generic phrasing like "is this video physically plausible?" As a result, the VLM judge's accuracy dropped directly to random guessing level.

This shows that scene-level binary QA is not a nice-to-have in the evaluation pipeline but a critical component of the entire protocol. When questions become vague, the judge's discriminative ability collapses—generalized plausibility perception cannot support specific physical inference, and this holds consistently across multiple VLMs.

An ablation study is a method used in machine learning research to identify whether a component is necessary: researchers deliberately remove or replace a part of the system and observe the performance change. If performance drops significantly after removal, the component is critical; if there is almost no effect, it may be redundant. PhyMetric's ablation replaces targeted binary questions with generalized plausibility questions, observing VLM judge performance while keeping other conditions unchanged. This "change only one variable" design gives the conclusion causal explanatory power rather than mere correlation. It is the methodological foundation for the paper's most striking finding and explains why the authors can assert that scene-level QA is a critical component of the protocol rather than an optional decoration.

Complementary Signals from Physical Priors

The role of structured physical priors is more subtle. Experiments show that priors can improve alignment between final scores and human judgment, but the magnitude of improvement fluctuates with VLM judge capability: stronger judges can better exploit the temporal sequences and observable signals in the priors, while weaker judges benefit only marginally and may even become unstable due to information overload.

This suggests a practical conclusion—physical priors are not a universal patch; their value depends on whether the downstream judge can digest this structured information. For benchmark designers, priors and judge capability need to be considered together.

In this context, a physical prior refers to structured prior knowledge about how a scene should evolve, including governing laws (such as the direction of gravitational acceleration), expected temporal sequences (such as the ball contacting the ground before bouncing), observable signals (such as deformation at the contact point), and common failure modes (such as penetration or hovering). Such priors are not learned from videos but injected manually or from a knowledge base, essentially telling the judge before evaluation what "correct" should look like. They are complementary to data-driven judgment: VLMs are good at identifying content from pixels but prone to errors in causal and temporal reasoning, and priors can provide constraints. However, injecting priors also comes at a cost—if the judge cannot effectively digest this structured information, it may instead degrade due to context overload, which explains the observed phenomenon that gains fluctuate with judge capability.

Empirical Results with Four Models and Three Judges

The authors conducted systematic experiments on four open-source T2V models and three VLM judges. The cross-model, cross-judge combinations validate PhyMetric's discriminative power: different models' strengths and weaknesses across physical categories are clearly separated, rather than compressed into a single broad ranking. This is precisely the core value of a diagnostic benchmark relative to leaderboard-style evaluation—it not only tells you whether something is good, but also where and why it is not.

The Next Stop for Physical Evaluation

Transforming physical plausibility from an unexplainable holistic score into a scene-by-scene, inspectable QA protocol is PhyMetric's main contribution. It reveals a more general evaluation principle: when the evaluation target itself contains complex structured constraints, the design of question phrasing is as important as model capability. For T2V researchers, this benchmark provides a reusable diagnostic tool; for evaluation methodology, it offers an example of moving from "scoring" to "diagnosis." The project page is publicly available at https://mengsiwei.github.io/PhyMetric/.

One-line Summary

PhyMetric reframes T2V physical plausibility evaluation as scene-level binary QA, enabling interpretable diagnosis.

Paragraph Summary

To address the "black box" problem in evaluating text-to-video model physical plausibility, PhyMetric proposes a diagnostic benchmark. It breaks each generated video into eight binary questions, covering 1,115 scenes, 50 physical categories, 8,920 answer-balanced questions, and 12 evaluation dimensions, with a vision-language model serving as judge answering each question and aggregating across multiple levels. The paper's key finding is that when targeted questions are replaced with a generic "is it physically plausible?" prompt, judge accuracy drops sharply to random level, proving that scene-level QA is core to the protocol rather than decoration. Structured physical priors can improve alignment with human judgment, but gains vary with judge capability. Experiments on four open-source T2V models and three VLM judges show that the benchmark can separate models' strengths and weaknesses across physical categories, advancing "scoring" into traceable "diagnosis."