Reading or Guessing? Urdu Font-Twin Benchmark Reveals the Truth About Multimodal OCR

A Font-Controlled Experiment That Cracks Open the "True Ability" of Multimodal OCR

When a multimodal model "reads" a passage of Urdu text, is it actually recognizing glyphs, or is it using language priors to "guess" a plausible answer? This question, which has long troubled OCR evaluation, was pushed to the foreground by a developer in a Kaggle Benchmarking Challenge using an elegant controlled experiment.

According to the project's original report published on dev.to, the researchers built an "Urdu Font-Twin Benchmark": the same NFC-normalized, SHA-256-identical Urdu Unicode string is rendered in two fonts — Nastaliq (Persian style, diagonal layout, dense glyphs, strongly context-dependent) and Naskh (more linear). If the model is truly reading glyphs, changing the font should not significantly change the transcription; if it is merely using language priors to fill in visual gaps, then when the text loses predictability, font differences should be amplified.

Nastaliq and Naskh are the two major calligraphic/typographic styles of the Arabic script system; both are used in Urdu, but their visual structures differ significantly. Nastaliq originated in Persia and is the most mainstream writing style for Urdu. Its characteristics include glyphs extending diagonally downward, characters within a word highly dependent on contextual shaping, an unstable horizontal baseline, and extremely complex ligatures and stacked structures — a word often appears as an inclined visual flow rather than a discrete column of characters. Naskh, by contrast, is more linear, with clear spacing and relatively independent character forms; it is the default style for Arabic typography and most digital fonts. For OCR, this difference means: in Nastaliq, the visual boundaries of individual glyphs are blurred, making it harder for models to segment and recognize character by character; while Naskh's high glyph separation makes character-by-character recognition more reliable. Precisely because of this, the same string has vastly different visual legibility across the two fonts, providing a natural control for distinguishing "reading glyphs" from "guessing words."

NFC normalization (Unicode Normalization Form C, i.e., Canonical Composition) is one of the four normalization forms defined by the Unicode standard. Its purpose is to unify equivalent character sequences into a canonical form. For example, certain characters can be represented either by a "base character + combining mark" or by a precomposed single code point; NFC preferentially chooses the latter (composed form). Performing NFC normalization in a text benchmark ensures that what is being compared is not surface differences arising from different encodings, but genuine glyph-recognition differences. The article emphasizes "NFC-normalized, SHA-256-identical Unicode strings," meaning the underlying character data rendered in the two fonts is byte-for-byte identical — any output difference can only come from visual rendering, not from encoding differences in the text input itself.

CER (Character Error Rate) is one of the most commonly used accuracy metrics in OCR and speech recognition. It is calculated by aligning the model output with the reference, counting the total number of characters in three error categories — substitutions (S), deletions (D), and insertions (I) — and dividing by the total number of characters in the reference: CER = (S + D + I) / N. A lower CER indicates more accurate transcription; 0 means perfectly correct. This article uses "paired Nastaliq CER minus Naskh CER" as the primary metric. This paired-difference design eliminates confounders such as text difficulty and overall model capability, directly measuring the performance change caused "solely by font difference." Similarly, WER (Word Error Rate) computes the same type of error rate at the word level; "pseudo-word WER" in the article refers to the word-by-word error rate between model output and reference under pseudo-text conditions.

Bootstrap confidence intervals are a non-parametric statistical method: by repeatedly resampling the sample with replacement (e.g., 1000 times) and computing the statistic for each resample, one obtains the empirical distribution and confidence interval of that statistic. When the sample size is small (this pilot has only 90 images and 45 pairs), bootstrap intervals can reflect estimation uncertainty more robustly than traditional normal approximations. "95% bootstrap interval [4.05, 12.82]" in the article means: if resampling were repeated, this CER gap would fall between 4.05 and 12.82 percentage points with 95% probability, and the interval does not contain zero, indicating the gap is statistically significant. For Claude Sonnet 5's -12.5 (interval [-33.5, 10.2]), the interval containing zero means the direction is opposite but not statistically significant.

Experimental Design: Three Language Conditions × Two Fonts

The benchmark includes three language conditions, progressively stripping away linguistic predictability:

• Natural text: Urdu Wikipedia sentences from a locked revision
• Shuffled text: the same vocabulary in a deterministic random order
• Pseudo-text: a deterministic "Urdu-like" character sequence preserving spaces, punctuation, and token length, but removing the predictability of ordinary vocabulary

Each item is rendered twice (once in Nastaliq, once in Naskh), and paired images share the same source text. The pilot set is 15 source items × 3 conditions × 2 fonts = 90 images, i.e., 45 exact text-font pairs.

Rendering Itself Is Part of the Experiment

A key insight from the researchers: if a font silently falls back, glyphs are clipped, or rendering sizes differ, the entire benchmark loses its meaning. Therefore, before calling the models, they used HarfBuzz to explicitly set the rtl, Arab, and ur properties to shape the string, then used FreeType to directly rasterize glyph IDs to prevent system font fallback, and verified character coverage, GSUB/GPOS tables, ligatures, RTL cluster order, etc. Offline validation passed all 90 images and 45 pairs, with no missing code points, fallbacks, or blank renders.

The primary scoring metric is paired Nastaliq CER minus Naskh CER. A positive value means the same source string produced more character errors in Nastaliq. Evaluation uses Unicode NFC and whitespace normalization, and does not merge easily confused Urdu and Arabic letters such as U+06CC/U+064A.

HarfBuzz is an open-source text shaping engine, widely used in Linux, Android, Chrome, and other systems. Its core responsibility is to convert Unicode strings into sequences with precise glyph IDs and positional information, handling: script identification, bidirectional text (RTL/LTR) ordering, context-dependent glyph substitution (via GSUB tables), glyph positioning (via GPOS tables, such as ligatures and kerning), etc. Setting the rtl, Arab, ur properties in the article explicitly tells HarfBuzz that this text is in the Arabic script system, is Urdu, and is written right-to-left, avoiding shaping errors caused by incorrect automatic detection. FreeType is the font rendering library responsible for rasterizing glyph outlines into bitmaps; "directly rasterizing glyph IDs with FreeType" means bypassing the operating system's font fallback mechanism, ensuring that the specified font is used for rendering rather than being silently replaced by another font because the system lacks a glyph. GSUB/GPOS tables are two core table types in OpenType font files: GSUB handles glyph substitution (such as ligatures and contextual variants), and GPOS handles glyph positioning (such as kerning and mark attachment). In Nastaliq, these tables are extremely complex because they carry the context-dependent glyph transformation logic. Offline validation of these tables ensures that font rendering does not produce incorrect glyph sequences due to missing tables or parsing errors.

Model Performance: A Multi-Model Picture More Interesting Than a Single Conclusion

The researchers queried Kaggle's catalog of 46 models, but the first expansion round exhausted the daily AI quota; ultimately, the paired analysis included 7 endpoints that completed all 90 image prompts: Claude Haiku 4.5/5.5, Claude Sonnet 5/5.5, Gemini 3.5 Flash-Lite, Gemini 3.7 Flash, and GPT-5.4 Mini. The authors explicitly state this is a validated pilot and an in-progress 46-model evaluation, not a completed general ranking.

Gemini 3.7 Flash was the most complete run. Its paired Nastaliq-minus-Naskh CER gaps were: +0.18 percentage points under natural conditions, +0.25 percentage points under shuffled conditions, and +8.75 percentage points under pseudo-text conditions (95% bootstrap interval [4.05, 12.82]). Its pseudo-word WER gap was +40.89 percentage points — the Nastaliq pseudo-word output introduced 9 natural vocabulary words not present in the reference, while Naskh introduced none.

At the multi-model level, pseudo-text CER gaps showed significant divergence: Claude Sonnet 5.5 at +35.9, Claude Haiku 4.5 at +21.8, Gemini 3.5 Flash-Lite at +12.9, GPT-5.4 Mini at +12.0, Gemini 3.7 Flash at +8.8, Claude Haiku 5.5 at +4.9 (interval contains zero), and Claude Sonnet 5 at -12.5 (interval [-33.5, 10.2], opposite direction). Among the 7 models with complete images, 5 had pseudo-word intervals excluding zero.

Language prior in this article's context refers to: multimodal models acquire extensive statistical patterns of natural language during training; when visual input is blurred or difficult to decipher, the model tends to output words that "look plausible" rather than faithfully transcribing the glyphs it sees. For example, when a spot in an Urdu text is blurred, the model might guess from context that it should be "کتاب" (book) rather than another visually similar but semantically nonsensical word. Language priors help the model complete and improve readability in natural text, but under pseudo-text conditions they become interference — because pseudo-text has no predictable semantic structure, and if the model still forcibly "guesses words," it will output natural vocabulary not present in the reference. The article's statement that "the pseudo-word output introduced 9 natural vocabulary words not present in the reference" is direct evidence of this mechanism: the model is not reading, it is guessing. The reason the paired Nastaliq-minus-Naskh CER gap is amplified under pseudo-text conditions is that Naskh's glyphs are clearer, making it easier for the model to transcribe faithfully; while Nastaliq's visual blur forces the model to rely more on language priors, and once the priors fail (pseudo-text), errors erupt in a concentrated way.

An Unexpected Finding: Font Is Not Just an "Appearance Format"

The researchers admit they expected pseudo-words to be harder, but did not anticipate that the gap between fonts would expand so dramatically for some models while remaining small or unstable for others. This conclusion points to a deeper judgment: font is not a surface-level input format choice; for some multimodal systems, it directly changes the nature of the task.

Several Claude runs also showed considerable gaps on natural or shuffled text, reminding researchers not to reduce all font effects to "language prior dependence." In addition, a visual "canary" test (a large number and a short Urdu word) revealed that model names appearing in the multimodal catalog do not guarantee that the endpoint can accept images, allocate enough visible output tokens, or follow precise transcription instructions. Reporting endpoint failures separately from OCR errors avoids disguising infrastructure behavior as accuracy scores.

Next Steps: Resolution Stratification and a Larger Source Set

Subsequent runs plan to complete retry-eligible endpoints, add 0.5× and 0.25× resolution stratification, and expand the source set from 15 to 60 items. The researchers also hope to add Tesseract Urdu and a public UrduOCR checkpoint, but only if executable model artifacts and their reuse licenses can be verified. The core analysis will remain paired: the question is not only which model has lower CER, but whether the same model reading the same string changes its result solely due to a change in Urdu font.

This experiment began with a typography question but ultimately touched on a broader proposition: when a vision-language model gives the correct word, how much of that answer comes from "seeing the letters," and how much from "knowing what word should be there"?

Full task, code, and reproducible resources are available in the Kaggle Benchmark (Reading or Guessing? Urdu Font Twins, version 9) and the GitHub repository syncaimain/urdu-font-twin-benchmark. The repository contains the deterministic renderer, source manifest, font licenses, canary tests, metric implementations, analysis scripts, tests, contact sheet, and a machine-readable status ledger.