MFVSR: A New Benchmark Brings Fine-Grained Evaluation to Video Subtitle Removal

Video Subtitle Removal (VSR) is a niche-sounding but high-frequency need in video restoration. Whether you're re-editing reposted content, localizing cross-border material, or restoring historical footage with hard-coded subtitles, you need to cleanly erase subtitles from the frame without damaging the underlying background. But for a long time, the evaluation standards for such methods have been surprisingly crude. The newly proposed MFVSR benchmark takes direct aim at this evaluation blind spot.

Why Existing Evaluation Falls Short

The paper points out that current VSR evaluation is constrained by three bottlenecks: low dataset resolution, fragmented and scattered test sets, and the practice of directly applying generic video inpainting protocols.

Generic inpainting protocols focus on whole-frame reconstruction quality, but the real difficulty of subtitle removal lies precisely in the local details—whether subtitle stroke edges leave residue, whether regions outside the subtitles get mistakenly altered, whether there's flickering across frames, and whether subtitles are missed or only partially removed. These fine-grained issues are almost entirely averaged away under whole-frame metrics. A method might perform well on whole-frame PSNR while the actual footage is riddled with residue and jitter.

In other words, past evaluations couldn't answer "is this method actually reliable?"—they could only produce a vague similarity score.

Video inpainting (video restoration/completion) refers to using spatiotemporal information in a video to fill in occluded, missing, or unwanted regions (such as subtitles, watermarks, scratches, or objects) with visually plausible background content, so that the restored video appears natural and coherent both within a single frame and across frames. Compared with image inpainting, video inpainting adds a temporal constraint: the same occluded region must maintain consistent texture, color, and structure across consecutive frames, otherwise the human eye easily detects flickering or jumps.

Generic video inpainting evaluation typically computes whole-frame reconstruction errors directly, such as PSNR, SSIM, or LPIPS, and treats these whole-frame metrics as quality scores. This protocol works acceptably for large-area missing-region reconstruction tasks, but it does not fit video subtitle removal: subtitles usually occupy only a small portion of the frame, so whole-frame metrics are "diluted" by the large amount of unmodified background pixels, and local residue, edge artifacts, and cross-frame jitter become nearly invisible after averaging. This is the context behind MFVSR's emphasis on spatial decoupling and fine-grained protocols.

Three Core Design Choices of MFVSR

MFVSR is positioned as a large-scale, multi-style benchmark for video subtitle removal oriented toward fine-grained and reliability-aware evaluation. Its distinguishing features center on three aspects: data, diversity, and evaluation protocol.

Data Scale and Sources

The benchmark contains 3,000 paired videos covering three source categories: Real-HQ (real high-quality), Real-LQ (real low-quality), and AI-Generated. In addition, there are 953 in-the-wild real videos as an auxiliary set for analyzing real-world coverage.

Paired videos mean each sample has both a "with-subtitle" and "without-subtitle" reference, which is the prerequisite for fine-grained evaluation. Including low-quality and AI-generated content also allows the benchmark to reflect robustness under different image quality conditions, rather than only scoring on ideal footage.

Controllable Subtitle Diversity

Subtitles themselves are far from uniform. MFVSR covers 16 languages and characterizes subtitle variation along four dimensions: spatial layout, temporal behavior, visual style, and linguistic pattern.

This design breaks "subtitles" down from a single variable into a controllable multi-factor combination. Horizontal versus vertical layout, static versus scrolling/fade-in-fade-out, outlined versus solid color, monolingual versus bilingual mixing—these pose entirely different challenges to removal algorithms. Only when the test set explicitly covers these dimensions can a method's capability boundaries be clearly seen.

Subtitle mask refers to a per-frame binary or soft-edge mask that annotates the region occupied by subtitle pixels, telling the algorithm "which locations need to be restored." In video subtitle removal, the presence or absence of a mask defines two fundamentally different task settings:

• mask-guided: the algorithm already knows the subtitle region and only needs to focus on cleanly restoring pixels within that area. This better reflects the upper bound of the restoration model itself.
• blind: the algorithm doesn't know where the subtitles are and must first detect or implicitly localize them before removing them. This examines both detection and restoration, and is closer to real deployment scenarios, where ready-made masks are often unavailable.

MFVSR evaluates these two settings separately, avoiding direct comparisons between mask-guided methods enjoying the advantage of a "known answer" and blind removal methods. It also lets researchers clearly see whether performance degradation comes from inaccurate localization or from incomplete restoration.

Information-Conditioned Spatially Decoupled Evaluation

This is the most methodologically valuable part of MFVSR. It introduces an information-conditioned, spatially decoupled evaluation protocol:

• distinguishing between blind (no subtitle mask provided) and mask-guided settings;
• separately measuring Restoration Fidelity (how cleanly the subtitle region is removed), Preservation Fidelity (whether non-subtitle regions are mistakenly altered), Temporal Fidelity (whether the result is stable across frames), coverage, and failures.

Separately measuring "did it restore correctly" and "did it damage anything else" is the key to this protocol. Traditional metrics conflate the two, so a method that removes cleanly but blurs the background and a method that is conservative but leaves obvious residue may end up with similar scores.

The core idea of spatially decoupled evaluation is to divide the frame into subtitle regions and non-subtitle regions and compute metrics separately, rather than averaging all frame pixels together.

Specifically, Restoration Fidelity measures reconstruction error only within subtitle regions, answering "is what should be removed removed cleanly?"; Preservation Fidelity measures only outside subtitle regions, answering "was anything that shouldn't be touched mistakenly altered?" Once separated, if a method over-diffuses inpainting to thoroughly remove subtitles and ends up blurring or distorting the background, its preservation fidelity will drop noticeably, whereas this loss might only appear as a tiny fluctuation after the decimal point in traditional whole-frame PSNR.

Temporal Fidelity compares restoration results of the same region across frames, specifically capturing flickering, color jumps, and texture instability. Combined with coverage and failure statistics, these three fidelity types can identify methods that "look good locally but are unreliable overall."

Findings from Six Methods

The paper experiments on six representative methods. The results reveal several groups of striking differences that are masked under traditional whole-frame metrics: there are clear inconsistencies among local restoration quality, out-of-region preservation quality, temporal consistency, and overall reliability.

This has direct practical implications. A method may restore the subtitle region very cleanly while introducing hard-to-detect changes in non-subtitle regions; or it may perform well per frame but flicker persistently across frames. Only a fine-grained protocol can expose the trade-offs among these dimensions.

What It Means for the VSR Field

The value of MFVSR is not just a larger test set. It tries to answer: what standard should we use to judge whether a subtitle removal method is "good"?

When evaluation shifts from whole-frame similarity to a spatially decoupled, temporally decoupled, reliability-aware fine-grained protocol, the optimization targets in method design shift accordingly. Researchers can more explicitly improve residue, mis-modification, and flickering separately, rather than vaguely chasing reconstruction metrics. For engineering scenarios that need to deploy subtitle removal, this diagnostic evaluation framework also provides a more real-world-relevant basis for method selection.

The benchmark paper itself doesn't propose a new method, but it redefines how this task should be measured—which is often the prerequisite for truly advancing a direction.

One-line Summary

MFVSR proposes a large-scale, multi-style benchmark for video subtitle removal, redefining reliable evaluation with a spatially decoupled protocol.

Paragraph Summary

MFVSR is a large-scale, multi-style benchmark for Video Subtitle Removal (VSR), designed to address existing bottlenecks such as low evaluation resolution, fragmented test sets, and the direct application of generic video inpainting protocols. It contains 3,000 paired videos spanning real high-quality, real low-quality, and AI-generated sources, plus 953 real videos as an auxiliary set, and covers 16 languages and four subtitle diversity dimensions: spatial layout, temporal behavior, visual style, and linguistic pattern. Methodologically, MFVSR proposes an information-conditioned spatially decoupled evaluation protocol that distinguishes blind and mask-guided settings, separately measuring restoration fidelity, preservation fidelity, temporal fidelity, coverage, and failures. Experiments on six representative methods reveal significant differences—masked by traditional whole-frame metrics—among local restoration quality, out-of-region preservation, temporal consistency, and overall reliability.