PhyAR from Shanghai Jiao Tong University: Solving the Semantic Prior Hijacking Problem in Video-LLM Physical Reasoning

The "Jumping to Conclusions" Problem When AI Watches Videos

Have you ever wondered how AI reacts when watching a video that violates the laws of physics? The answer might surprise you — not only does it fail to flag the anomaly, it confidently explains "this is perfectly normal."

A recent study from Shanghai Jiao Tong University reveals a core deficiency in current Video-LLMs (Video Large Language Models): the model isn't blind to physical anomalies — it's "overthinking." Semantic priors act like an invisible hijacker, overpowering the model's visual perception and causing it to override objective facts with subjective expectations. The research team proposes the PACC dataset and VARC (Visually Anchored Reasoning Chain) mechanism, which enables AI to "see" physical reality again without modifying the model architecture.

What is a Video-LLM? A Video-LLM is an extension of multimodal large language models into the video understanding domain. Its core architecture typically consists of three components: a Vision Encoder (such as CLIP or ViT) that extracts visual features from video frames; a Projector that maps visual features into the language model's semantic space; and an LLM Backbone responsible for cross-modal reasoning and text generation. Representative models include VideoLLaMA, Video-ChatGPT, GPT-4V, and others. These models excel at video question answering and video captioning, but their reasoning mechanisms are highly dependent on statistical patterns accumulated from training data — which is precisely the root cause of the semantic prior problem.

How Semantic Priors Hijack Physical Reasoning

The Nature of Dual Failure

The paper reveals the essence of the problem through three elegantly designed experiments. In counter-physical and counter-intuitive scenarios, Video-LLMs produce erroneous reasoning due to logical or statistical priors, completely ignoring objective visual evidence. Even more surprising, when researchers attempted to inject physical knowledge into the model, erroneous rationalization actually intensified — the model became better at "fabricating reasons" to explain obviously anomalous phenomena.

However, through single-frame spatial relationship description experiments, researchers demonstrated that the model's low-level perception capabilities are actually intact. This rules out the "perceptual deficit" hypothesis, pinpointing the root cause as semantic priors hijacking the reasoning process.

The Deep Mechanism of Semantic Priors Semantic priors originate from the Bayesian inference framework, referring to the "preset expectations" that models form based on statistical patterns accumulated from training data when processing new inputs. In large language models, these priors are deeply encoded into model weights through joint training on massive text and image data. For example, after seeing millions of image-text pairs of "cup falls → shatters," the model forms a strong causal prior. When visual input conflicts with this prior, the model tends to "believe" the prior rather than the visual evidence — similar to "Confirmation Bias" in cognitive science. Semantic priors are not inherently a defect; they significantly improve reasoning efficiency in normal scenarios, but become a fatal cognitive blind spot in counter-physical or edge-case scenarios.

Unified Attribution Theory

The paper presents a key insight for the first time: the model's severe errors in counter-intuitive scenarios and its failures in counter-physical scenarios are fundamentally homologous. Whether violating objective physical laws (such as objects disappearing into thin air) or violating statistical expectations (such as a cup falling without breaking), the root cause of model failure is the inertia of semantic priors overwhelming the objectivity of visual perception. The prior acts as a "perception bypass," directly paralyzing the model's ability to anchor physical facts.

PACC Dataset: Precise Diagnosis of Physical Reasoning Deficiencies

Construction Pipeline and Design Philosophy

The PACC dataset adopts a Human-in-the-Loop (HITL) paradigm, divided into four rigorous stages:

• Stages 1 and 2: Screening positive samples and performing visual fact anchoring, combining AI generation with expert review to eliminate any subjective speculation and retain only pure visual facts
• Stages 3 and 4: Adversarial generation and dual-expert review, synthesizing negative samples through manual CV editing or AI generation, and generating Code labels

Why Choose the HITL Paradigm? Human-in-the-Loop is a methodology that embeds human expert judgment into AI system training or evaluation workflows, widely adopted in data annotation, model alignment, and quality control. Compared to purely automated annotation, HITL can effectively handle ambiguous boundary cases requiring professional judgment, but at the cost of significantly increased data construction time and expense. In AI safety and alignment, RLHF (Reinforcement Learning from Human Feedback) is the most representative application of HITL. PACC adopts the HITL paradigm precisely because violations of physical laws often require domain experts to accurately judge — purely automated annotation easily introduces noise, causing models to learn incorrect judgment criteria and ultimately degenerating into "artifact detectors" that rely on pixel-level artifacts.

The final dataset strictly decouples generation artifacts from genuine physical fallacies, preventing the model from relying on pixel-level shortcuts and degenerating into an "artifact detector," forcing it to make judgments based on deep physical logic.

Dual-Stream Classification System

PACC contains counter-physical and counter-intuitive dual streams, covering 8 fine-grained physical fallacy dimensions:

• Counter-Physical Stream: Targeting rationalization fallacies, constructing high-fidelity scenarios that violate objective laws, such as coherence violation, causal inversion, object disappearance, and identity mutation
• Counter-Intuitive Stream: Targeting stereotypical fallacies, constructing scenarios that violate statistical expectations but are physically plausible, such as near-miss and consequence termination

Although the dataset currently contains only 758 sample pairs due to rigorous HITL expert verification — a relatively limited scale — this design prioritizes high fidelity and physical rigor, making it a precise diagnostic benchmark for physical reasoning.

VARC Mechanism: Reasoning Chain Reconstruction That Cuts Semantic Shortcuts

Three-Step Forced Reasoning Pipeline

VARC restructures the reasoning process as a directed Markov chain, cutting semantic shortcuts unrelated to visual input, comprising three mandatory steps:

1. Forced Observation: Extract explicit visual evidence (e.g., shadow changes), blocking hallucinations driven by semantic completion
2. Causal Attribution: Perform logical conflict detection based on objective observations and physical knowledge, such as determining whether shadow absence violates optical laws
3. Evidence Adjudication: The final verdict is strictly constrained to the preceding observation and attribution nodes

The Reasoning Significance of Directed Markov Chains A directed Markov chain describes a "memoryless" state transition process in probability theory: the current state depends only on