TinyLlama-1.1B
A small language model based on the LLaMA architecture with 1.1B parameters, commonly used for research and applications in resource-constrained settings.
Timeline (last 90 days)
MuRIL is an encoder with an Indian-language vocabulary, and its parameter count is only one-fifth of TinyLlama's.
A study based on a corpus containing 131,000 sentences covering 11 Indian scripts compared the tokenizer performance of models such as TinyLlama-1.1B, mBERT, and MuRIL.
TinyLlama-1.1B's tokenizer produces an average of 1.34 subword tokens per word when processing Latin script, and an average of 22.60 tokens per word when processing Kannada.
Within a 2048-token context window, English users can obtain 1528 words of usable context, while Kannada users can obtain only 91 words, a difference of nearly 17 times.
In TinyLlama's vocabulary, the codepoint coverage is 0% for Odia, 1% for Telugu, and 2% for Kannada.
Among the 11 Indian scripts, 5 scripts reach the upper limit of UTF-8 byte fallback fertility, Odia reaches 99.6% of the upper limit, and 93.6% of its tokens are raw bytes.
TinyLlama's UNK rate is 0.0% on all tested scripts, because SentencePiece's byte fallback mechanism never produces UNK tokens.
Running a 4-bit quantized version of TinyLlama-1.1B requires at least approximately 600 MB of memory, exceeding the hardware ceiling of most consumer-grade routers.
All Facts (8)
MuRIL is an encoder with an Indian-language vocabulary, and its parameter count is only one-fifth of TinyLlama's.
50%UnverifiedA study based on a corpus containing 131,000 sentences covering 11 Indian scripts compared the tokenizer performance of models such as TinyLlama-1.1B, mBERT, and MuRIL.
50%UnverifiedTinyLlama-1.1B's tokenizer produces an average of 1.34 subword tokens per word when processing Latin script, and an average of 22.60 tokens per word when processing Kannada.
50%UnverifiedWithin a 2048-token context window, English users can obtain 1528 words of usable context, while Kannada users can obtain only 91 words, a difference of nearly 17 times.
50%UnverifiedIn TinyLlama's vocabulary, the codepoint coverage is 0% for Odia, 1% for Telugu, and 2% for Kannada.
50%UnverifiedAmong the 11 Indian scripts, 5 scripts reach the upper limit of UTF-8 byte fallback fertility, Odia reaches 99.6% of the upper limit, and 93.6% of its tokens are raw bytes.
50%UnverifiedTinyLlama's UNK rate is 0.0% on all tested scripts, because SentencePiece's byte fallback mechanism never produces UNK tokens.
50%UnverifiedRunning a 4-bit quantized version of TinyLlama-1.1B requires at least approximately 600 MB of memory, exceeding the hardware ceiling of most consumer-grade routers.
50%