Unverified50% confidenceFactExact time
Tokka-Bench proposes five evaluation dimensions: bytes per token, unique token coverage, fertility, word split rate, and vocabulary composition.
1
Sources
50%
Confidence
Long-term
Relevance
10/8/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedTokka-Bench is an open-source multi-metric tokenizer evaluation framework covering 100 natural languages and 20 programming languages.62% similarUnverifiedDolch词汇表包含面向幼儿的500个核心词汇，可用于约束语言模型输出61% similarUnverifiedThe tool's "Grammar Analysis" feature uses levels such as L0 and L1 to annotate the nesting relationships between main clauses and subordinate clauses, and identifies the subject, predicate, and object at each level.61% similarUnverifiedVADER通过五条启发式规则处理标点符号、大写字母、程度副词、连词but的转折语义以及否定词60% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/986632API
curl https://kongchang.com/api/v1/knowledge/claims/986632MCP
get_claim(id=986632)