FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Video Streams?

Streaming Video LLMs have become a hot topic in multimodal research in recent years, capable of continuously understanding visual content within a continuous video stream. However, existing benchmarks almost all focus on low-dynamic scenes—slow shot changes, sparse events, and stable backgrounds. When the scene shifts to real-world high-dynamic streams (fast motion, sudden actions, dense interactions), how do these models actually perform? The newly proposed FastBench delivers a rather pessimistic answer.

Background: Sparse Sampling Misses More Than Just Frames

Streaming VLMs face a fundamental dilemma: the context budget is limited, yet video streams are infinitely long. Under this constraint, models must simultaneously balance three dimensions—temporal history length, spatial resolution, and temporal granularity.

A common engineering practice is sparse sampling at 1–2 FPS. This significantly compresses token consumption but comes at the cost of directly losing fast events. An action completed in 0.3 seconds, an instantaneous state switch—these simply won't appear in sampled frames at 2 FPS. Previously, there was no benchmark specifically targeting this failure mode—FastBench was designed to fill this gap.

Streaming Video LLMs differ fundamentally from traditional offline video understanding models. Offline models typically obtain the complete video first, then perform global sampling and encoding, allowing repeated review of any segment; streaming models, however, face continuously arriving video streams with no endpoint and must make online judgments at each moment based on a limited context window, without knowing what comes next. This "causality" constraint means models cannot retrospectively retrieve key frames as in offline scenarios—they can only rely on historical information already cached at that moment. Consequently, sampling strategies, history compression methods, and temporal granularity allocation all directly determine whether a model can capture fleeting events. The high-dynamic scenarios that FastBench focuses on are precisely the test conditions that most amplify this dilemma.

How FastBench Constructs a High-Dynamic Evaluation Set

The core of FastBench is a trajectory-grounded data pipeline, rather than simply randomly extracting frames from videos to create questions. Its process consists of four steps:

1. Generate QA from high-frame-rate segments: Construct questions based on high-FPS footage, ensuring the questions themselves involve fast dynamic information;
2. Filter out questions answerable at low frame rates: Remove questions that can still be answered correctly at 2 FPS, retaining only samples that truly depend on high temporal granularity;
3. Use SAM3 and CoTracker3 for trajectory answer verification: Leverage segmentation and point-tracking models to automatically validate answers and evidence intervals;
4. Three rounds of manual inspection: Conduct human review after automated verification to control annotation quality.

The final dataset contains 306 QA pairs, covering eight domains and six capabilities, divided by temporal reference into forward, instant, and backward time ranges. Each sample includes manually annotated evidence intervals, meaning the evaluation not only judges whether answers are correct but can also locate whether the model has attended to the correct temporal segments.

SAM3 (Segment Anything Model 3) is a new-generation version of Meta's Segment Anything model series, supporting open-vocabulary segmentation based on text prompts and video object tracking, capable of continuously producing segmentation masks for targets in videos. CoTracker3 is a model focused on dense point tracking, capable of long-term trajectory prediction for any pixel in a video, maintaining good continuity even under occlusion or fast motion. Their division of labor in the FastBench pipeline is: SAM3 provides object-level segmentation and identity consistency, while CoTracker3 provides pixel-level motion trajectories. Combined, they can automatically verify whether a QA's answer corresponds to a genuinely occurring dynamic event and provide the corresponding time interval. This "trajectory-grounded" verification is more reliable than relying solely on human effort or pure text heuristic rules, and also provides traceable evidence for subsequent manual review.

ProactiveFrame: Letting the Model Decide "How Dense to Look"

Beyond the evaluation set, the authors also propose a training-free baseline called ProactiveFrame. The idea is straightforward: rather than passively receiving at a fixed frame rate, let the model actively adjust the input frame rate through text tokens—increasing sampling density when fine temporal perception is needed, and maintaining sparsity at other times.

The accompanying dual-layer sliding window mechanism manages historical information: recent frames retain high-FPS observations, while older frames are downsampled into sparse history, thereby maintaining a memory structure of "fine nearby, coarse distant" within a limited context. This design can be integrated into existing streaming architectures without fine-tuning, representing a typical inference-time strategy optimization.

Training-free refers to not modifying model parameters or performing additional fine-tuning, but rather changing model behavior through strategy adjustments only at the inference stage. The greatest advantage of such methods is plug-and-play capability: any existing streaming VLM architecture can directly adopt them without retraining or annotated data. ProactiveFrame leverages exactly this—it allows the model to "request" higher sampling density by generating specific text tokens, essentially turning frame rate control into part of the language generation task. The dual-layer sliding window is a memory management structure: the recent portion within the window maintains high-frame-rate observations to capture details, while older history outside the window is downsampled into sparse representations to save context. This "dense near, sparse far" design mimics the temporal gradient characteristics of human memory and is a common yet effective engineering trade-off in long video stream understanding.

Experimental Results: Even the Strongest Model Gets Only Half Right

The most striking finding on FastBench is the low performance ceiling: the best-performing model, Gemini-3.5-Flash, scores only 50.7%, indicating that current streaming VLMs are still far from meeting the mark in high-dynamic perception.

Several further observations are worth noting:

• Higher frame rates help, but with a ceiling: Qwen3-VL-8B improves from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains gradually saturate as history gets compressed. This means simply piling on sampling rates is not a sustainable solution.
• ProactiveFrame outperforms uniform sparse sampling: It improves by 5.4 and 1.5 percentage points respectively over sparse uniform sampling, proving that "on-demand densification" does yield benefits.
• But falls far short of oracle guidance: There remains a significant gap between ProactiveFrame and human/oracle-style focusing. In other words, models struggle to infer from the video stream alone that "this moment requires finer temporal perception."

The last point is especially critical: it shifts the problem from "insufficient compute" to "insufficient metacognition." What the model lacks is not just frames, but the ability to judge when to request more frames.

Where High-Dynamic Streaming Understanding Goes Next

The value of FastBench is not just a leaderboard, but exposing a long-neglected dimension in streaming VLM evaluation. It suggests several directions worth exploring:

• Adaptive temporal sampling should move from fixed strategies to content-driven approaches, with ProactiveFrame as a trainable starting point;
• Evidence interval supervision can serve as a training signal to help models learn to locate critical moments;
• Joint optimization of context compression and temporal granularity, rather than treating them as independent hyperparameters to tune.

Code and data are open-sourced on GitHub (Ashone3/FastBench). For teams working on video understanding, multimodal agents, and streaming inference, this benchmark provides a testbed for checking whether their methods truly "see fast."

One-line Summary

FastBench reveals the bottleneck in high-dynamic streaming video understanding: the strongest model achieves only 50% accuracy, with the problem lying in insufficient metacognition.

Paragraph Summary

The article introduces FastBench, a streaming video LLM evaluation benchmark for high-dynamic real-world video streams. Existing benchmarks mostly focus on low-dynamic scenes and cannot expose the problem of sparse sampling missing fast events. FastBench employs a trajectory-grounded pipeline, combining SAM3 and CoTracker3 for answer verification, with three rounds of manual inspection, constructing a dataset of 306 QA pairs with evidence interval annotations. Experiments show the strongest model, Gemini-3.5-Flash, scores only 50.7%, while Qwen3-VL-8B's gains saturate when scaling from 2 FPS to 24 FPS. The authors' training-free baseline ProactiveFrame actively adjusts frame rates through text tokens, outperforming sparse uniform sampling but falling far short of oracle guidance. The core conclusion: the bottleneck is not compute, but the model's difficulty in judging when finer temporal perception is needed.