Datalab Open-Sources OmniParseBench: 16,288 OCR Tests to Audit Document Parsers

OCR document parser evaluation has long faced an awkward situation: vendors publish wildly different accuracy numbers, and both the evaluation sets and judging criteria are often opaque, making it hard for developers to determine how a given model will actually perform on their own business documents. Datalab's open-source benchmark OmniParseBench aims to change this — by turning evaluation from "scoring" into "question-answering."

From Scoring to Question-Answering: A Shift in Evaluation Philosophy

The core design of OmniParseBench is 16,288 yes/no tests spanning 2,937 pages and 92 languages. Each test item is essentially a true-or-false question: does the parser's output satisfy a specific condition? The final score is not a fuzzy similarity metric but a pass rate — a percentage that can be understood and compared directly.

This design solves several pain points in traditional OCR evaluation. Metrics based on edit distance or character accuracy tend to mask structural errors; a misrecognized digit and an entire misaligned table might produce similar scores. The question-answering approach breaks evaluation down into specific issues, making failure modes visible: was the footer missed, were two columns of text joined into one line, or were superscripts and subscripts in a math formula lost?

For developers, this means results can be sliced by tag, script, or language. The parsing problems of an Arabic RTL document and those of a vertically typeset Chinese classical text won't be lumped into a single overall score, making troubleshooting far clearer.

Edit distance and character accuracy are the two most commonly used metrics in traditional OCR evaluation. Edit distance (Levenshtein Distance) measures the minimum number of insertions, deletions, and substitutions needed to transform the recognition output into the ground truth, which is then converted into character-level or word-level accuracy. The advantage of such metrics is that they are simple to compute and language-agnostic, but their drawbacks are obvious: they are insensitive to the "location" and "structure" of errors. For example, if an entire column of numbers in a table is shifted by one column, or if header text is mixed into the body, the character-level difference may be small and the similarity score remains high; conversely, a single punctuation difference and an entire missing paragraph may be statistically averaged out. More importantly, edit distance produces a continuous score lacking a clear semantic boundary of "pass/fail." Differences in preprocessing and postprocessing between vendors can significantly affect the final number, distorting cross-vendor comparisons. The question-answering approach directly addresses this flaw by replacing continuous scores with attributable, discrete judgments.

What 92 Languages and Multi-Script Coverage Means

The coverage of 92 languages is a distinguishing feature of OmniParseBench compared to most similar benchmarks. Mainstream OCR evaluation sets are typically English-dominant, supplemented by a handful of European languages, with quite limited coverage of low-resource languages and non-Latin writing systems.

The practical value of multi-script coverage lies in exposing model language bias. A parser that performs excellently on English documents may exhibit systematic character confusion or layout comprehension errors on Devanagari, Arabic, or Cyrillic. OmniParseBench's pass rates sliced by script can directly reveal such biases, so developers are no longer misled by impressive numbers in a single language when selecting a model.

It's worth noting that increased language coverage also means the number of test pages per language is relatively dispersed (2,937 pages spread across 92 languages), so the sample size for some low-resource languages may be limited. In practice, it's advisable to check the sample count under a specific language slice and avoid over-interpreting small samples.

Script and "language" are concepts on two different dimensions, and understanding this is key to interpreting OmniParseBench's sliced results. A single language may use multiple scripts — for example, Kazakh uses both Cyrillic and Latin, and Serbian uses both Cyrillic and Latin; conversely, a single script is shared by multiple languages, such as Arabic being used by Arabic, Persian, Urdu, and others. For OCR, the real technical difficulties often lie at the script level rather than the language level: Arabic and Hebrew are RTL (right-to-left) and letters change form depending on position, Devanagari has consonant cluster ligatures, Thai has no spaces between words, and Chinese and Japanese involve vertical typesetting and large character sets. Therefore, pass rates sliced by script reveal systematic weaknesses in glyph recognition, ligature handling, and layout direction more effectively than slicing by language alone.

Sliceable Pass Rates: Making Auditing Truly Usable

"Auditable" is a keyword in OmniParseBench's positioning. Traditional benchmarks often provide only a single leaderboard score, and users cannot probe what lies behind the number. Every score in OmniParseBench can be drilled down by tag, script, or language, bringing changes at two levels.

For model developers, the distribution of failed items points directly to improvement directions. If the pass rate is notably low on table-heavy pages while nearly perfect on plain-text pages, the optimization focus becomes clear. For enterprise adopters, slicing makes "is this model suitable for my use case" a verifiable question — if your business documents are concentrated in Japanese invoices and Korean forms, you only need to focus on the corresponding slices' pass rates.

The open-source nature further reinforces audit value: evaluation sets, test items, and judging logic are all inspectable. Vendors cannot cherry-pick favorable evaluation subsets to beautify results, and third parties can reproduce and verify them.

Benchmark reproducibility is the technical prerequisite for "auditability" to be achievable. Even if an evaluation set is open-source, if it lacks a fixed data version, explicit prompts or parsing configurations, and a deterministic scoring script, third parties still struggle to obtain scores consistent with the vendor's. OmniParseBench opens both test items and judging logic together, meaning anyone can independently run the same inputs and reach the same conclusions. This is especially important for OCR tasks, which are heavily affected by image preprocessing, resolution, and model version — the same model with a different image scaling or binarization strategy may yield drifting results. Reproducibility is also key to preventing "benchmark contamination": once an evaluation set is public, models may be trained specifically against it, so benchmarks need continuous updates or private holdout sets to maintain their credibility as a selection basis over the long term.

Potential Impact on the OCR Ecosystem

OmniParseBench shifts the competitive dimension of OCR evaluation from "who scores higher" to "who is reliable in specific scenarios." When pass rates can be sliced along any dimension, the cost of claiming "industry-leading" becomes higher — it must withstand cross-validation across multiple languages, scripts, and layouts.

For downstream application developers, such benchmarks reduce information asymmetry in model selection. In the past, evaluating a document parsing API often required annotating your own data and building an evaluation pipeline; now you can use OmniParseBench's public results for initial screening and focus your energy on validating business-specific documents.

No single benchmark can solve all problems. Factors such as layout complexity of real business documents, scan quality, and handwritten content still need to be verified in specific scenarios. But a transparent, sliceable, question-answering open-source benchmark at least moves OCR evaluation one step from "trust the vendor" toward "look at the evidence."

One-line Summary

Datalab open-sources OmniParseBench, an OCR benchmark replacing scoring with question-answering and supporting multilingual slice auditing.

Paragraph Summary

Datalab's open-source benchmark OmniParseBench aims to change the opaque, inconsistent standards of OCR evaluation. It contains 16,288 yes/no questions spanning 2,937 pages and 92 languages, presenting scores as pass rates rather than fuzzy similarities. This "question-answering" approach makes structural failure modes visible — missed footers, merged columns, lost formula superscripts — and enables troubleshooting by tag, script, or language. Coverage of 92 languages and multiple scripts exposes model language bias, but sample sizes for some low-resource languages are limited and should be interpreted cautiously. Sliceability and open-source auditability make it hard for vendors to cherry-pick favorable subsets, allow third-party reproduction, and push OCR evaluation from "trust the vendor" toward "look at the evidence."