Sarvam-M
A large language model developed by Sarvam AI with a focus on Indian languages. Its tokenizer provides relatively good coverage of Indian languages.
Timeline (last 90 days)
For the same Hindi sentence using different tokenizers, the token cost multipliers are: Qwen3/Qwen2.5 4.44 times, Sarvam-M 1.79 times, and Gemma-3 1.31 times.
The Qwen series vocabulary is mainly Chinese and English with limited subword coverage for Indian languages, whereas Gemma-3 and Sarvam-M included more Indian language data in training and therefore segment more efficiently.
All Facts (2)
For the same Hindi sentence using different tokenizers, the token cost multipliers are: Qwen3/Qwen2.5 4.44 times, Sarvam-M 1.79 times, and Gemma-3 1.31 times.
50%UnverifiedThe Qwen series vocabulary is mainly Chinese and English with limited subword coverage for Indian languages, whereas Gemma-3 and Sarvam-M included more Indian language data in training and therefore segment more efficiently.
50%