Unverified50% confidenceFactExact time
Measured with the Qwen3 tokenizer on FLORES devtest sentences, Hindi consumes 4.44 times as many tokens per sentence as English.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedUnder the Qwen3 tokenizer, Malayalam averages 13.47 tokens per word, while English averages 1.27 tokens per word.79% similarUnverifiedThe passive vocabulary of adult native speakers is typically 3–5 times larger than their active vocabulary.71% similarUnverifiedThe Qwen series vocabulary is mainly Chinese and English with limited subword coverage for Indian languages, whereas Gemma-3 and Sarvam-M included more Indian language data in training and therefore segment more efficiently.69% similarUnverifiedUnder naive byte-level tokenization, Bengali is compressed to an average of approximately 1.4 characters per token, compared to approximately 3.9 for English, causing the linguistic balance of the corpus to be inverted.67% similarUnverifiedIn a 16-segment professional translation test (full score 48), Qwen 9B scored 43 with direct translation, 45 with added preceding context, 47 with a glossary added, and 47 with both preceding context and glossary added.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1067567API
curl https://kongchang.com/api/v1/knowledge/claims/1067567MCP
get_claim(id=1067567)