P³IBench: Benchmarking Proactive Video Assistants from Perception to Intervention

When Video Assistants Learn to Be "Proactive": The Birth of a New Benchmark

Multimodal large models have made rapid progress in video understanding, but most existing benchmarks care about only one thing: whether a model can understand what has already happened on screen. This premise itself limits the imagination—a real-world assistant should not be merely a passive description tool; it also needs to judge what the user will encounter next and whether it should speak up at this moment.

To address this gap, researchers propose P³IBench, a benchmark for proactive video understanding in real streaming scenarios. It breaks "proactivity" down into three complementary capabilities: daily perception, risk prediction, and intervention. In other words, the model must not only describe the scene, but also anticipate risks and decide whether and when to act.

Three Capabilities, Three Tests

Daily Perception: Don't Take Common Sense for Granted

Daily perception examines a model's understanding of ordinary events in a continuous video stream—for example, what is being cooked in a kitchen, or the movement intent of pedestrians on a street. This part seems basic, but streaming input means the model must continuously update its judgment when information is incomplete, rather than waiting until the video ends to reason all at once. For non-streaming systems accustomed to offline processing, this is itself a barrier.

The difference between streaming and non-streaming processing is the key premise for understanding this benchmark. A non-streaming system can typically access the entire video, first extracting all frames and then encoding and reasoning over them uniformly—equivalent to "watch first, answer later." A streaming system, by contrast, can only receive frame-by-frame or short-segment input in temporal order, and is required to output judgments at any moment. This introduces two additional constraints: first, causality, meaning the model cannot use information from future frames; second, state maintenance, meaning the model must compress historical information into a limited context or memory state and keep updating it as new frames arrive. As a result, the same model that performs well in an offline setting may degrade significantly under a streaming setting—what is being evaluated is the ability to "think while watching," not mere visual recognition.

Interactive video QA (streaming / interactive video QA) is a related task paradigm in this direction. Its difference from P³IBench is that the former is usually triggered by an explicit user question, and the model only needs to ensure accuracy in passive answering; the latter requires the model to decide on its own whether to output, even without an explicit instruction, thereby making "response decision-making" itself an object of evaluation.

Risk Prediction: Saying What May Happen Before It Happens

Risk prediction requires the model to infer the future based on the current frame—whether a pedestrian ahead will suddenly cross, whether a pot on the stove is about to burn dry. The core difficulty of such tasks is not object recognition, but translating visual cues into an actionable time window. Predicting too early seems superfluous; predicting too late is worthless.

Intervention: Knowing When to Speak and When to Stay Silent

Intervention capability is closest to the word "assistant." The model needs to output reminders or suggestions at the right moment, rather than commenting on every frame. The paper particularly emphasizes a dimension that is easy to overlook: appropriate restraint. A system that constantly raises alarms is no better than a silent one; calibrated response decision-making is the true embodiment of proactivity.

Calibration of response decisions here refers to the degree of match between the model's confidence in the binary judgment of "whether it should act" and its actual correctness rate. A well-calibrated system will speak only when it is truly confident and the situation genuinely requires a reminder; conversely, an overly sensitive system will produce many false alarms, and users will quickly learn to ignore it—this is what is commonly called "alarm fatigue." In the context of proactive assistants, the cost of false positives is often higher than that of false negatives: an unnecessary interruption continuously erodes user trust, while the consequences of a missed alert depend on the specific scenario.

Evaluating such capabilities commonly uses combinations of precision and recall—the former measures how many of the alerts raised were necessary, while the latter measures how many moments that truly should have been flagged were captured. Their trade-off curves (such as F1 or PR-AUC) better reflect the balance between "restraint" and "timeliness" than a single accuracy metric. The appropriate restraint emphasized in the paper essentially requires the model to land in a reasonable range on this trade-off, rather than blindly pursuing recall.

Running 12 Models Exposes the Same Problem

The study evaluated 12 representative open-source and proprietary video-language models, including both streaming systems and adapted non-streaming systems. The results are not encouraging: current models generally struggle to make well-calibrated response decisions.

In plain terms—they either fail to act when they should, or frequently speak when they should not interrupt. Models with strong visual understanding are not necessarily better at "when to respond," suggesting that the bottleneck for proactivity is not only at the perception layer. This also explains why the paper's title places perception, prediction, and intervention on the same chain: all three are indispensable, but the latter two are clearly the current weaknesses.

Training a 4B Assistant: How Much of the Gap Can It Close?

The authors further constructed additional proactive training data and trained P³I-Assistant-4B. This assistant with 4 billion parameters achieved improvements on multiple tasks, but the paper admits that the gains across different capabilities are uneven.

This unevenness itself is informative. If adding data led to comprehensive improvement, the problem would simply be insufficient data; but in reality, some capabilities improve markedly while others stagnate, suggesting that proactive video understanding may require more specialized designs in architecture or training objectives, rather than simply feeding general video data to a larger model.

The "4B" in P³I-Assistant-4B means the model has about 4 billion parameters. In video-language models, parameter scale is not simply positively correlated with proactive capability: larger models are usually stronger at perception and common-sense reasoning, but "when to respond" is a temporal decision problem, often more strongly influenced by training objectives and data distribution. This is also why the paper finds uneven gains—using a small model specifically for proactive training can approach or even surpass general large models on specific tasks, because learning signals for proactive weights are scarce, and scale advantages cannot automatically compensate for that.

This "small but specialized" route has appeared repeatedly in recent multimodal research: by constructing training data with temporal annotations (when to speak, what to say), proactive decision-making is explicitly modeled as a supervised objective, rather than expecting the model to acquire it on its own from general video description data. Its limitations are also obvious—the coverage of specialized data determines the boundaries of capability, which is why follow-up work needs to continue advancing data diversity and scale.

The Next Step for Proactive Assistants Is Not Just "Understanding Better"

The value of P³IBench lies in advancing the evaluation focus from "what the model sees" to "what the model should do and when to do it." Its conclusion is clear: effective proactive assistance requires not only visual understanding, but also precise judgment of response timing and appropriate restraint.

For future research, this means two things. First, evaluation systems need to cover decision quality along the temporal dimension, not just recognition accuracy; second, model design needs to explicitly model the action of "whether to respond," treating it as an independent learning objective. As video assistants move from description tools to true collaborators, judgment may be more important than sight.

One-line Summary

P³IBench advances video understanding evaluation toward proactive assistance, revealing a widespread weakness in models' judgment of response timing.

Paragraph Summary

Multimodal large models are advancing rapidly in video understanding, but existing evaluations mostly stop at "understanding what has already happened on screen." Researchers propose P³IBench, evaluating proactive video understanding in real streaming scenarios and decomposing proactivity into three complementary capabilities: daily perception, risk prediction, and intervention. It requires models not only to describe scenes, but also to anticipate risks and decide whether and when to remind. Evaluation of 12 representative open-source and proprietary video-language models shows that current models generally struggle to make well-calibrated response decisions: either failing to act when they should, or frequently speaking when they should not interrupt, and strong visual understanding does not equal accurate judgment of response timing. The authors further train P³I-Assistant-4B with proactive training data, achieving improvements on multiple tasks, but the gains across capabilities are uneven, suggesting that proactive video understanding requires specialized design in architecture or training objectives, not just more data or scale.