Unverified50% confidenceFactExact time
A study based on a corpus containing 131,000 sentences covering 11 Indian scripts compared the tokenizer performance of models such as TinyLlama-1.1B, mBERT, and MuRIL.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
Unverified最新的大语言模型支持12.8万到100万token的上下文，但处理数十万字长篇小说时仍会出现信息遗忘和前后矛盾71% similarUnverifiedThe evaluation subjects of the ContextAdapt study cover 12 mainstream large language models.69% similarVerifiedThe vocabulary of a standard autoregressive language model typically contains 32,000 to 128,000 tokens.68% similarUnverifiedTinyLlama-1.1B's tokenizer produces an average of 1.34 subword tokens per word when processing Latin script, and an average of 22.60 tokens per word when processing Kannada.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1074682API
curl https://kongchang.com/api/v1/knowledge/claims/1074682MCP
get_claim(id=1074682)