Fertility Rate (Tokenizer)

A core NLP metric for evaluating the multilingual friendliness of tokenizers, calculated as the total number of tokens produced after tokenizing a target-language corpus divided by the total number of original words. Higher values indicate more fragmented segmentation; an efficient range is generally 1.0–1.5.