PhyMetric
A diagnostic benchmark for text-to-video generation models that assesses physical plausibility. It uses scene-level binary question answering to evaluate whether generated videos obey physical laws, covering dimensions such as gravity, collision, deformation, fluids and lighting.
Timeline (last 90 days)
PhyMetric is a diagnostic benchmark that reframes the evaluation of physical plausibility in text-to-video (T2V) generation as verifiable scene-level question answering.
PhyMetric contains 1,115 scenes spanning 6 domains and 50 physical categories.
PhyMetric is equipped with 8,920 answer-balanced binary questions distributed across 12 evaluation dimensions.
Each scene in PhyMetric is paired with eight binary (yes/no) questions covering physical dimensions such as gravity, collision, deformation, fluids, and lighting.
PhyMetric's evaluation is performed by a VLM as the judge, answering on question-aware sampled frames, after which accuracy is aggregated at four levels: scene, dimension, physical category, and model.
Each scene in PhyMetric comes with structured physical priors containing four pieces of information: the physical laws governing the scene, the expected temporal sequence, observable feature signals, and common failure modes.
When targeted binary questions are replaced with generic prompts of the form "is this video physically plausible?", the VLM judge's accuracy drops to the level of random guessing.
Scene-level binary question answering is a key component of PhyMetric's evaluation protocol rather than an optional decoration; generalized plausibility perception cannot carry specific physical inference.
The authors conducted systematic experiments on four open-source T2V models and three VLM judges, validating the discriminative power of PhyMetric.
The PhyMetric project page has been made public at https://mengsiwei.github.io/PhyMetric/.
All Facts (10)
PhyMetric is a diagnostic benchmark that reframes the evaluation of physical plausibility in text-to-video (T2V) generation as verifiable scene-level question answering.
50%UnverifiedPhyMetric contains 1,115 scenes spanning 6 domains and 50 physical categories.
50%UnverifiedPhyMetric is equipped with 8,920 answer-balanced binary questions distributed across 12 evaluation dimensions.
50%UnverifiedEach scene in PhyMetric is paired with eight binary (yes/no) questions covering physical dimensions such as gravity, collision, deformation, fluids, and lighting.
50%UnverifiedPhyMetric's evaluation is performed by a VLM as the judge, answering on question-aware sampled frames, after which accuracy is aggregated at four levels: scene, dimension, physical category, and model.
50%UnverifiedEach scene in PhyMetric comes with structured physical priors containing four pieces of information: the physical laws governing the scene, the expected temporal sequence, observable feature signals, and common failure modes.
50%UnverifiedWhen targeted binary questions are replaced with generic prompts of the form "is this video physically plausible?", the VLM judge's accuracy drops to the level of random guessing.
50%UnverifiedScene-level binary question answering is a key component of PhyMetric's evaluation protocol rather than an optional decoration; generalized plausibility perception cannot carry specific physical inference.
50%UnverifiedThe authors conducted systematic experiments on four open-source T2V models and three VLM judges, validating the discriminative power of PhyMetric.
50%UnverifiedThe PhyMetric project page has been made public at https://mengsiwei.github.io/PhyMetric/.
50%