VCR-Bench: An Open-Source Framework Filling the Gap in Video Classification Robustness Evaluation

Why Video Classification Has Long Lacked a Unified Robustness Standard

Image classification has long had mature robustness benchmarks, allowing researchers to compare models' ability to withstand adversarial attacks under a unified protocol. But video classification has remained fragmented — with no widely accepted, standardized benchmark.

The root cause lies in the additional temporal dimension of video data. Compared to static images, video introduces extra degrees of freedom along the time axis, which directly affects the design of adversarial attacks, defenses, and preprocessing pipelines. Temporal sampling strategies, perturbation budget allocation, and how evaluation metrics are aggregated across frames are all intertwined factors that have no direct counterparts in image-based scenarios.

As a result, robustness research on video classifiers is scattered across mutually incompatible implementations. Numbers reported in different papers are difficult to reproduce and even harder to compare, leaving the field without a reliable reference point.

Adversarial attacks are techniques that introduce imperceptible perturbations to input data, causing machine learning models to make incorrect predictions. In image classification, these perturbations typically appear as pixel-level noise constrained by L∞ or L2 norms (i.e., an upper bound on perturbation magnitude). In video settings, attackers must decide not only how much perturbation to apply per frame, but also how to distribute it across the temporal axis — whether to apply it uniformly across all frames, concentrate it on key frames, or exploit inter-frame correlations to craft temporally consistent perturbations. Defense strategies include adversarial training (exposing the model to adversarial examples during training to improve robustness) and input preprocessing (such as denoising or randomization). While standardized implementations of these attack-and-defense mechanisms are abundant in the image domain, how to adapt them to video remains contested and lacks consensus — which is the fundamental technical reason why video robustness evaluation is so difficult to compare across studies.

What VCR-Bench Does

VCR-Bench is positioned as a modular, open-source benchmark framework aimed at standardizing every stage of video classification robustness evaluation. It covers the full pipeline from data to results:

• Video loading: Unified data reading to eliminate result discrepancies caused by loading differences
• Classifier wrappers: A standardized interface for invoking different models
• Adversarial attacks and defenses: Built-in attack and defense modules
• Perceptual quality metrics: Measuring the impact of perturbations on visual quality
• Configuration presets and result logging: Ensuring reproducibility of experiments

This modular design means researchers no longer need to build evaluation pipelines from scratch — instead, they can plug in and swap different components under the same protocol for fair comparison.

Integration Scale

VCR-Bench has already integrated a substantial component library:

• 30 video classification models
• 14 adversarial attack methods
• 10 defense wrappers

All of these operate under a shared evaluation protocol — exactly the kind of foundational infrastructure that has been missing from the field.

Evaluation Dimensions and Experimental Setup

The research team evaluated representative video classifiers, attack methods, and defenses on a subset of the Kinetics-400 dataset. Notably, VCR-Bench does not report a single accuracy metric — it assesses performance across multiple dimensions:

• Clean accuracy: Baseline model performance without any attacks
• Attack success rate: Measuring the effectiveness of attacks
• Perceptual quality: Whether adversarial perturbations are sufficiently imperceptible
• Runtime: Computational cost of each method
• Memory usage: Resource consumption

The significance of this multi-dimensional evaluation is that robustness is never an isolated number. An attack method may have a high success rate, but if its perturbations are visually noticeable or require massive computational resources, its value within a realistic threat model is diminished. Only by examining accuracy, perceptual quality, and resource overhead together can we get a true picture of the tradeoffs on both the attack and defense sides.

Kinetics-400 is a large-scale video action recognition dataset released by DeepMind, containing approximately 300,000 video clips sourced from YouTube across 400 human action categories (such as sports, cooking, and playing musical instruments), with each clip lasting about 10 seconds. It is one of the most widely used benchmarks for pretraining and evaluation in video understanding, and many mainstream video classification models (such as TSN, SlowFast, and VideoSwin) use it as a standard evaluation set. Using a subset of this dataset for robustness evaluation ensures comparability with existing literature while reducing the computational cost of large-scale attack experiments. That said, Kinetics-400 primarily focuses on action recognition scenarios, and its generalizability to vertical domains such as surveillance, medical imaging, and autonomous driving remains to be validated — which is the background behind the "limited scenario coverage" mentioned at the end of this article.

Reproducibility as a Core Value

VCR-Bench's most notable contribution may not be the number of models or attacks it integrates, but rather its systematic commitment to reproducibility. The framework comes with detailed installation documentation, reproducible run presets, component extension interfaces, and scripts for reproducing the paper's results. The project is open-sourced on GitHub.

In current AI research, the difficulty of reproduction is a widespread pain point — and the video domain is particularly affected, given the large data volumes, complex preprocessing, and multiple sources of randomness. By fixing configuration presets and unifying logging, VCR-Bench makes "the reported numbers can be reproduced by others" the default rather than a luxury.

The component extension interface is equally important: it means the benchmark is not a one-off static artifact, but can continuously evolve as new models, attacks, and defenses emerge. The community can contribute new components without undermining the existing basis for comparison.

Implications for Video Robustness Research

The emergence of VCR-Bench has the potential to move video classification robustness research from "scattered and incomparable" to "standardized and cumulative." Only when researchers use the same protocol and the same set of metrics can progress in the field be genuinely quantified and tracked.

For practitioners concerned with video AI security, the value of this kind of infrastructure is often underestimated. It's not as eye-catching as a new model, but it defines the scale by which subsequent work is evaluated. Whose attack is stronger, whose defense is more robust — these questions ultimately need to be settled on a commonly accepted arena, and VCR-Bench is trying to become exactly that.

Of course, the current evaluation is limited to a subset of Kinetics-400, and the datasets and scenarios covered remain constrained. The benchmark's true impact will also depend on community adoption and ongoing maintenance. But as a first step toward filling a long-standing gap, its direction is clear and pragmatic.