TextVBench: A New Benchmark for Evaluating Text Rendering in Video Generation

Why the "Text Problem" in Video Generation Is Hard to Measure

Modern video generation models can synthesize convincingly realistic scenes, yet they consistently stumble when it comes to text in the frame: misspelling characters, omitting text specified in the prompt, or letting the same words subtly shift appearance across frames. For use cases that rely on on-screen text to convey information — advertising, signage, captions, brand placements — these failures are critical.

The challenge is that measuring such failures is far harder than it sounds. The most intuitive approach is to run OCR on each frame independently, but when a scene contains multiple text targets and OCR returns several noisy detection regions, the evaluator immediately faces an attribution problem: which recognized result corresponds to which piece of text that was "supposed to appear"? This is the core problem TextVBench sets out to solve.

TextVBench Reframes Evaluation as a Spatiotemporal Correspondence Problem

TextVBench's key innovation is formalizing visual text evaluation as a multi-target, multi-instance spatiotemporal correspondence problem. Rather than simply "scoring whatever gets recognized," it first establishes a mapping between expected text and actual observations, then scores based on that mapping.

This approach leads to three concrete design differences:

• Target-conditioned candidate selection: Instead of asking a vision-language model (VLM) to perform unconstrained free transcription, TextVBench filters candidate regions given the known target text, significantly reducing false positives.
• One-to-many mapping support: In practice, a single target string may be split or appear multiple times across OCR regions; the framework allows one target to correspond to multiple recognized results.
• Spatial and temporal fidelity treated separately: Frame-level spatial fidelity (is the text correct in a single frame?) and cross-frame temporal fidelity (does the text remain consistent over time?) are measured independently, preventing the two failure modes from collapsing into a single undifferentiated score.

Two Complementary Evaluation Tracks

TextVBench defines two complementary tracks:

• TextVBench-T2V: Measures a model's ability to create and sustain specified text from a text prompt — evaluating "generation from scratch" quality.
• TextVBench-Ref2V: Measures a model's ability to preserve existing text from a reference image — evaluating "fidelity to source" quality.

The two tracks cover the two main categories of text-handling tasks in video generation, making evaluation results more practically actionable.

Spatiotemporal correspondence is a classical problem in computer vision, originally used to describe feature matching of the same object across different frames and viewpoints. TextVBench imports this concept into text evaluation, meaning the scoring system must not only determine "whether the correct text appears in a given frame" but also track "whether the same piece of text maintains consistent identity across time." This modeling approach naturally handles two common failure modes: text appearing in unexpected positions (spatial drift), and text content quietly shifting between frames (temporal instability). Traditional per-frame OCR can only detect the former; it is nearly blind to the latter, since it treats each frame as an independent snapshot with no cross-frame linkage.

The T2V/Ref2V distinction maps onto two fundamentally different commercial needs. A typical T2V scenario is advertising creative generation: given a tagline, the model must naturally render the corresponding text within the video. Ref2V scenarios arise more often in product videos or branded content: a reference image already contains a trademark, logo, or label, and the generated video must faithfully preserve those visual elements. The latter is inherently harder because the model must not only "write" the correct characters, but also maintain pixel-level consistency with the reference image's typography, color, and layout — placing much higher demands on fine-grained controllability.

A Traceable Scoring System with Diagnostic Capability

TextVBench uses a length-weighted harmonic score to combine these multiple factors. The harmonic mean is sensitive to weak links — if any single dimension performs poorly, the overall score is pulled down noticeably — making it a more honest reflection of real-world usability than simple weighted averaging.

Beyond the main score, the framework also provides extra-text diagnostics, specifically designed to surface failure patterns hidden by a single OCR score — for example, cases where a model hallucinates text that was never requested in the prompt.

Crucially, the entire benchmark is organized around controlled attributes: text content, layout, carrier, and motion. This means an aggregate score can be traced back to specific capability bottlenecks — whether the failure stems from the text content itself, the typography, the carrier object, or motion blur can all be pinpointed precisely.

The harmonic mean differs from the arithmetic mean in its penalty for extreme low values: if precision is 1 and recall is 0, the arithmetic mean still yields 0.5, while the harmonic mean collapses to zero. TextVBench uses a length-weighted variant that further assigns weights by target string character length, making the difficulty difference between generating "Hello" and generating "OPEN 24 HOURS" explicit. This design prevents occasional hits on short text from masking systematic failures on long text, and makes evaluation results more reflective of model performance in real commercial scenarios such as advertising slogans or product serial numbers.

Benchmark Results: More Reference Text, Worse Performance

Among models evaluated, MiniMax-H3 achieved a TextVScore of 68.32 on the T2V track, the strongest result; the top score on the Ref2V track was 56.51. The roughly ten-point gap between tracks indicates that "preserving existing text from a reference image" is harder than "generating text from a prompt" — which aligns with intuition, since fidelity tasks impose stricter demands on detail consistency.

Attribute-slice analysis reveals a clear additional pattern: scores drop significantly as the number of co-occurring reference text regions in a scene increases. This shows that current video generation models still have a pronounced weakness in complex scenes with multiple simultaneous text targets — they can manage a single text element, but when targets multiply, performance degrades noticeably.

Implications for the Industry

As video generation models evolve from "capable of moving" to "fit for use," accurate text rendering is becoming a critical threshold for commercial deployment. TextVBench offers a more rigorous evaluation paradigm than per-frame OCR, transforming the vague question of "does the text look right" into a quantifiable, traceable, bottleneck-localizable metric system.

For researchers, its attribute-slice design clarifies where improvements should be targeted. For engineering teams, the T2V/Ref2V distinction helps with model selection for specific business scenarios. As purpose-built benchmarks like this gain wider adoption, progress in video generation model text handling will become both more measurable and more attainable.