Unverified50% confidenceFactExact time
The Unigram language model tokenization method was proposed by Kudo in 2018, starting from a large vocabulary and progressively pruning it to retain the set of subwords that minimizes the likelihood loss on the corpus.
1
Sources
50%
Confidence
Long-term
Relevance
9/30/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedLarge language models process language by splitting input text into tokens, which can be complete words, sub-word fragments, or individual characters.64% similarUnverifiedTraditional n-gram models and neural language models trained on a single language assign extremely low probability scores to cross-lingual word sequences, causing the decoder to tend to forcibly map foreign-language words to phonetically similar words in the target language.64% similarUnverifiedFor low-resource languages with explicit orthographic rules, standardized phonemic orthography allows fundamental tasks such as tokenization and sentence segmentation to be accomplished through deterministic rules, without relying on data-hungry deep learning models.64% similarUnverified后处理纠错词典方案通过维护音译拼写到正确术语的映射表，在识别结果输出后进行字符串替换，实现简单但需人工维护且对新术语无能为力63% similarUnverifiedNon-standard registers tend to omit function words (such as articles and copulas), making sentences conveying the same meaning literally shorter, and thus appearing simpler under length-guided complexity estimation.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/963023API
curl https://kongchang.com/api/v1/knowledge/claims/963023MCP
get_claim(id=963023)