FastBench
An evaluation benchmark for high-dynamics video understanding aimed at streaming video large language models. It contains 306 QA pairs covering eight domains and six capabilities, is constructed via a trajectory-driven data pipeline, and includes human-annotated evidence intervals.
Timeline (last 90 days)
Existing streaming video large language model evaluation benchmarks almost all focus on low-dynamic scenes, lacking dedicated evaluation for high-dynamic scenes.
FastBench is a newly proposed benchmark for high-dynamic streaming video understanding, with a dataset containing 306 QA pairs, covering eight domains and six capabilities.
FastBench is constructed using a trajectory-driven data pipeline, comprising four steps: generating QA from high-frame-rate clips, filtering out questions answerable at low frame rate, using SAM3 and CoTracker3 for trajectory answer verification, and three rounds of manual inspection.
Each sample in FastBench comes with a manually annotated evidence interval, and the evaluation not only judges whether the answer is correct but also can locate whether the model attended to the correct time segment.
On FastBench, the best-performing model, Gemini-3.5-Flash, achieves only 50.7% accuracy.
On FastBench, Qwen3-VL-8B improves from 32.9% at 2 FPS to 44.6% at 24 FPS, but as history is compressed, the gains gradually saturate.
The code and data of FastBench are open-sourced on GitHub (repository path Ashone3/FastBench).
All Facts (7)
Each sample in FastBench comes with a manually annotated evidence interval, and the evaluation not only judges whether the answer is correct but also can locate whether the model attended to the correct time segment.
50%UnverifiedOn FastBench, the best-performing model, Gemini-3.5-Flash, achieves only 50.7% accuracy.
50%UnverifiedOn FastBench, Qwen3-VL-8B improves from 32.9% at 2 FPS to 44.6% at 24 FPS, but as history is compressed, the gains gradually saturate.
50%UnverifiedFastBench is constructed using a trajectory-driven data pipeline, comprising four steps: generating QA from high-frame-rate clips, filtering out questions answerable at low frame rate, using SAM3 and CoTracker3 for trajectory answer verification, and three rounds of manual inspection.
50%UnverifiedExisting streaming video large language model evaluation benchmarks almost all focus on low-dynamic scenes, lacking dedicated evaluation for high-dynamic scenes.
50%UnverifiedFastBench is a newly proposed benchmark for high-dynamic streaming video understanding, with a dataset containing 306 QA pairs, covering eight domains and six capabilities.
50%UnverifiedThe code and data of FastBench are open-sourced on GitHub (repository path Ashone3/FastBench).
50%