MorphScore
An evaluation metric that measures the consistency between tokenization results and human-annotated true morpheme boundaries, used to assess tokenization quality from the perspective of linguistic morphological structure.
Timeline (last 90 days)
LCT achieves lower fertility and higher MorphScore on 104 languages, while the cross-language tokenization cost differences remain similar.
MorphScore measures the degree of consistency between segmentation results and linguistic morphological structure by comparing tokenization boundaries with manually annotated true morpheme boundaries.
All Facts (2)
LCT achieves lower fertility and higher MorphScore on 104 languages, while the cross-language tokenization cost differences remain similar.
50%UnverifiedMorphScore measures the degree of consistency between segmentation results and linguistic morphological structure by comparing tokenization boundaries with manually annotated true morpheme boundaries.
50%