New Benchmark for Sindhi Tokenizers: How SentencePiece Unigram Outperforms 14 Existing Solutions

First unified benchmark evaluation of Sindhi tokenizers: the new 64k model slightly outperforms existing baselines and reveals silent character deletion risks.
For Sindhi, a low-resource Indo-Aryan language, the authors trained SentencePiece Unigram tokenizers at four vocabulary sizes from 32k to 150k based on approximately 1.1 million multi-source sentences, and systematically compared them with 14 existing tokenizers across four evaluation sets: FLORES+, Wikipedia, sentiment corpus, and news corpus. The 64k configuration averages 1.147 tokens per word, outperforming the best baseline's 1.208 and XLM-RoBERTa's 1.512, retains 88.6% of words as single tokens, and leads overall on fertility, single-token retention rate, and UNK rate. Round-trip checks also found up to 0.85% silent character deletion in two existing tokenizers. However, the authors candidly acknowledge the small gap with the best baseline, lack of significance testing, and no evaluation of downstream task impacts.
Low-resource language tokenizers have long been in a state of "each going their own way"—many models have been released, yet few studies conduct cross-comparisons on unified data. A recent paper on Sindhi fills this gap: the authors trained a tokenizer based on SentencePiece Unigram and systematically compared it with 14 existing tokenizers across four evaluation sets.

Why Sindhi Tokenization Deserves Dedicated Study
Sindhi is a low-resource Indo-Aryan language, primarily written using an extended Perso-Arabic script. Although multiple Sindhi tokenizers have been released on public model hubs, they have never been compared against each other on shared data. This "no-benchmark" state makes it difficult for researchers and developers to determine which tokenizer truly suits their tasks.
The paper's authors collected approximately 1.1 million Sindhi sentences from four public sources, and added a smaller, recurring slice of Urdu, Persian, and Arabic, training four SentencePiece Unigram tokenizers at vocabulary sizes ranging from 32k to 150k, all with byte fallback enabled.
SentencePiece Unigram is a tokenization algorithm proposed by Google, one of the two main models supported by the SentencePiece toolkit (the other being BPE). Unlike BPE, which bottom-up merges high-frequency character pairs, the Unigram model starts from a large vocabulary and iteratively prunes subwords that contribute least to the corpus likelihood via the Expectation-Maximization (EM) algorithm, until the target vocabulary size is reached. This "top-down" approach naturally supports probabilistic tokenization—that is, multiple segmentations can be sampled for the same text (used as regularization during training), while at inference the highest-probability segmentation is chosen. Unigram is especially suited to morphologically rich agglutinative languages and low-resource languages, because it can more flexibly balance vocabulary coverage and compression efficiency. The paper chose Unigram over BPE precisely for its fine-grained control over subword granularity under limited corpora.
Byte fallback is a safety mechanism provided by SentencePiece: when the input text contains characters not in the vocabulary, the tokenizer does not map them to UNK (unknown token), but instead splits the character's UTF-8 byte sequence into several byte-level tokens for output. This way any Unicode character can be represented losslessly, fundamentally eliminating character-level information loss. For Sindhi, which uses the extended Perso-Arabic script, this mechanism is particularly important, because the script contains numerous diacritics and special glyphs that are difficult to fully cover in a vocabulary. With byte fallback enabled, the UNK rate can be reduced to zero, but the cost is that rare characters are split into multiple byte tokens, potentially lowering compression efficiency. All new tokenizers in the paper enable this mechanism, which is also the key reason their round-trip character loss is zero.
Evaluation Design: Four Datasets, Five Metrics
The comparative experiments were conducted on four evaluation sets: FLORES+, Sindhi Wikipedia, a news-derived sentiment corpus, and a contemporary news corpus. Evaluation metrics include fertility (tokens per word), single-token retention rate, UNK rate, byte-fallback rate, and round-trip character loss.
These metrics cover the core capabilities of a tokenizer: compression efficiency (fertility), vocabulary coverage (single-token retention rate), unknown word handling (UNK rate), and character-level safety (round-trip loss).
Fertility (tokens per word) is the core metric for measuring a tokenizer's compression efficiency, calculated as the total number of tokens after tokenization divided by the number of words in the original text (usually split by whitespace). The lower the fertility, the fewer subwords each word is on average split into, and the shorter the sequence length required for the model to process the same text, thereby reducing computational overhead and mitigating long-distance dependency issues. For low-resource languages, excessively high fertility shortens the effective context window and reduces training sample efficiency. The 64k new tokenizer in the paper has a fertility of 1.147, meaning each Sindhi word on average requires only about 1.15 tokens, approaching the ideal state of "one token per word." Note that there is a trade-off between fertility and vocabulary size: larger vocabularies generally yield lower fertility, but the embedding matrix and output layer also grow, making overfitting easy in low-resource scenarios.
Performance at 64k Vocabulary Size
At the 64k vocabulary setting, the new tokenizer averages 1.147 tokens per word, outperforming the best existing tokenizer's 1.208 and XLM-RoBERTa's 1.512. Meanwhile, 88.6% of words are retained as single tokens.
On the three metrics of fertility, single-token retention rate, and UNK rate, the 64k and above configurations are best or tied for best across all four evaluation sets. However, at the 32k configuration, it lags behind one baseline model on FLORES+.
The authors also candidly point out that the gap with the best Sindhi-adapted baseline is small—the fertility gap at 64k on FLORES+ is only 0.002, and no significance testing was reported. This means the advantage exists but is not overwhelming.
Character Safety: Hidden Issues Revealed by Round-Trip Checks
A noteworthy finding comes from the round-trip character loss check. The new tokenizer has no character loss, but two existing tokenizers exhibit silent character deletion, deleting up to 0.85% of characters; another seven tokenizers have character loss below 0.1%.
Silent character deletion is especially dangerous for low-resource languages, because character-level information loss can be amplified in downstream tasks, and users often find it hard to detect.
Round-trip character loss refers to the proportion of characters lost when tokenizing the original text and then decoding it back to a string, and is a direct means of testing a tokenizer's character-level safety. Ideally, tokenization should satisfy "encoding-decoding" identity, meaning the text after round-trip is completely identical to the original. However, when a tokenizer uses UNK markers to handle unseen characters, or silently deletes certain characters during normalization, the round-trip result deviates from the original. This loss is usually negligible in Latin-script languages, but can be significant in low-resource languages using complex scripts. The paper found that two existing tokenizers silently deleted up to 0.85% of characters—such problems do not raise errors and are hard to notice, yet they systematically contaminate training data and are amplified in downstream tasks. Round-trip checks therefore become an indispensable line of defense when evaluating low-resource tokenizers.
Limitations and Open Questions
The authors explicitly state that the evaluation is limited to intrinsic metrics, and the impact on downstream tasks has not been tested. Moreover, the small gap with the best baseline and the lack of significance testing mean that the "best" conclusion should be viewed with caution. The code and tokenizer files have been made public (DOI: 10.5281/zenodo.23962474), providing a foundation for subsequent reproduction and downstream evaluation.
For Sindhi NLP practitioners, the value of this work lies not only in providing a stronger tokenizer option, but also in establishing a reproducible evaluation framework—exactly the infrastructure that is scarcest in low-resource language research.
One-line Summary
First unified benchmark evaluation of Sindhi tokenizers: the new 64k model slightly outperforms existing baselines and reveals hidden silent character deletion issues.
Paragraph Summary
For Sindhi, a low-resource Indo-Aryan language, the paper's authors trained SentencePiece Unigram tokenizers at four vocabulary sizes from 32k to 150k based on approximately 1.1 million multi-source sentences, and systematically compared them with 14 existing tokenizers across four evaluation sets: FLORES+, Wikipedia, sentiment corpus, and news corpus. The 64k configuration averages 1.147 tokens per word, outperforming the best baseline's 1.208 and XLM-RoBERTa's 1.512, retains 88.6% of words as single tokens, and leads overall on fertility, single-token retention rate, and UNK rate. Round-trip checks also found that two existing tokenizers exhibit up to 0.85% silent character deletion. However, the authors candidly acknowledge that the gap with the best baseline is small, significance testing is lacking, and downstream task impacts were not evaluated.
Related articles

Gemini 4 Neon Hands-On Test: Best for Coding but Nearly Twice as Slow — How Do You Account for That?
Overseas reviewers tested Gemini 4 Neon: ranked first on the Coding Agent Index with per-task cost about half of Claude, but nearly twice as slow as the median. We break down third-party cost, speed, and capability data to help you decide if the bill is worth it.

Gemini 4 Neon Hands-On: How Does It Compare to Astra 6.2 for PPT Generation?
Hands-on test of Google Gemini 4 Neon vs Astra 6.2 on the same offline speech PPT task: visuals on par, Neon wins on accuracy and detail and faithfully converts the data source; but revisions return only single pages, and UI and images are easily redrawn. The fast model matches the flagship and is worth using.

Google Gemini 4 Neon Released: Three-Tier Model System and Pricing Breakdown
Google releases Gemini 4 Neon, positioned between Hydrogen and Carbon for complex high-frequency tasks with multimodal input. Artificial Analysis composite intelligence index around 60, per-task cost about 30% lower than Carbon. Also: Grok 4.8 multi-version testing, Anthropic IPO priced at $83.