AskStream: A New Paradigm for Proactive Questioning in Streaming Video Understanding

Multimodal large language models are changing the way video understanding is interacted with. From offline processing to streaming responses to proactive interaction, models are evolving from passive answerers into proactive agents capable of deciding when to answer. However, existing proactive paradigms generally assume that the information needed to answer a query is complete, and the model only needs to choose between waiting and answering. In reality, uncertainty from the video or the user often leaves information incomplete, trapping the model in endless waiting or causing severe hallucinations. To address this pain point, researchers propose treating questioning as a third proactive action and introduce AskStream, the first multi-turn benchmark that systematically evaluates models' ability to decide when to ask and what to ask under uncertainty.

From Waiting to Asking: The Third Leap in Proactive Interaction

Traditional video understanding models follow an offline paradigm: they receive a complete video and output an answer. Streaming understanding advances this to the online setting, where the model must decide within a video stream when to respond. Proactive interaction further grants the model the decision rights to wait or answer, bringing it closer to a real assistant. But the paper points out that this binary decision framework carries a strong implicit assumption: the required information will eventually appear in full. When video content is blurry or occluded, or when the user's query itself is ambiguous, a model that only waits may never get an answer, while a model that forces an answer is prone to hallucination.

The authors therefore argue for redefining proactive interaction as three actions: wait, ask, and answer. The ask action lets the model proactively seek missing information from the user, thereby advancing the dialogue amid uncertainty. This shift upgrades the model from an information receiver to an information collaborator, better matching the reality that information is often incomplete.

Streaming video understanding refers to online inference as video data continuously arrives, rather than waiting for the entire video to be processed. Unlike the offline paradigm, a streaming model must make judgments at any moment about the segment observed so far, and thus faces the problem of partial observability: subsequent frames may contain critical information, or they may never appear. This requires temporal decision-making—dynamically assessing information sufficiency based on accumulated evidence. Proactive interaction extends this capability to the level of whether to respond, and the three-action framework further requires the model to proactively query the user when information is insufficient, turning a partially observable problem into an interactively solvable collaborative process.

The AskStream Benchmark: Quantifying When to Ask and What to Ask

To study how models should ask before answering, the research team built AskStream, the first multi-turn streaming video understanding benchmark covering both video uncertainty and user uncertainty. The benchmark contains four task categories and eight subtasks, spans 438 videos, and includes manual annotations of 2,714 frame-level time windows, each corresponding to one of three action types: wait, ask, or answer. This fine-grained annotation moves evaluation beyond final-answer correctness and into the quality of decision timing and question content.

For evaluation, the authors set up both multiple-choice and open-ended settings and propose the Ask-F1 metric, which jointly measures model performance on when to ask and what to ask. Experimental results show that even the strongest model reaches only 52.18% on this metric, exposing a significant deficiency in current models' proactive information-seeking ability under uncertainty.

Ask-F1 is a composite metric designed in the paper to evaluate proactive questioning strategies. Its core idea is to decompose when to ask and what to ask into quantifiable matching problems. Specifically, the benchmark annotates each frame-level time window with a ground-truth action (wait/ask/answer) and the key information points that a question should cover. Ask-F1 measures, along the temporal dimension, how well the model's chosen questioning time overlaps with the annotated window, and along the content dimension, whether the model's question covers the missing key information; the two are combined via a harmonic mean. Unlike traditional answer accuracy, this metric can penalize both failing to ask when one should and asking when one should not, thereby reflecting the quality of the model's interaction strategy under uncertainty more comprehensively.

Sources of Uncertainty and Task Design

AskStream divides uncertainty into two types: video uncertainty and user uncertainty. Video uncertainty arises from missing visual information such as blurry frames, occluded targets, or events that have not yet occurred; user uncertainty comes from query ambiguity, unclear references, or vague intent. The benchmark's four task categories are organized around these two types of uncertainty and require the model to dynamically judge, during streaming, whether the currently available information is sufficient to answer.

For example, when a target has not yet clearly appeared in the video, the model should choose to wait; when the user's question lacks a key reference, the model should proactively ask for clarification; only when information is sufficient should the model answer. This three-action framework expands traditional answer accuracy evaluation into a comprehensive assessment of interaction strategy.

User uncertainty is fundamentally different from video uncertainty: the latter stems from the absence of objective visual evidence, while the former stems from ambiguity in the natural-language query itself. For example, the referent in that thing he just mentioned may depend on dialogue history or context knowledge shared with the user and cannot be resolved from video frames alone. In streaming settings, user uncertainty also manifests as intent drift over time—users may revise or refine their questions while the video plays. This requires the model not only to sense whether visual information is sufficient but also to reason about the semantic completeness of the query and decide whether to clarify by asking or to wait for more context. AskStream annotates the two types of uncertainty separately, enabling evaluation to pinpoint which type of uncertainty the model is weaker at.

Performance Bottlenecks and Future Directions

An Ask-F1 score of 52.18% means existing models still have considerable room for improvement in proactive questioning. The paper's analysis notes that models tend to act too early or too late when judging when to ask, and often pose irrelevant or redundant questions when deciding what to ask. This reflects both insufficient perception and reasoning about uncertainty in current multimodal models and a lack of training data targeting questioning strategies.

AskStream provides a quantifiable research foundation for this direction. Future work may proceed along three lines: building larger-scale multi-turn questioning training data, designing uncertainty-aware reward mechanisms, and integrating questioning ability into end-to-end streaming video understanding systems. For practical applications, a video assistant capable of proactive questioning has clear value in scenarios such as remote assistance, accessibility description, and intelligent surveillance.

Conclusion

AskStream's core contribution is formally incorporating questioning into the proactive interaction framework for streaming video understanding and using a fine-grained benchmark to reveal current models' shortcomings. From the binary decision of waiting versus answering to the ternary strategy of waiting, asking, and answering, this shift is not only a technical extension but also a response to the reality that information is often incomplete in real interaction. Although the strongest model's Ask-F1 of only 52.18% indicates a long road ahead, the establishment of the benchmark provides a clear yardstick for future research.