Vocabulary Richness
The ratio of the number of distinct word types to the total number of words in a text, directly reflecting lexical diversity; it is a commonly used metric for evaluating output degeneration in language models.
Timeline (last 90 days)
As α→0, Renyi entropy approaches the observed vocabulary richness (the logarithm of the number of word types that have appeared); at α=1 it equals Shannon entropy; at α=2 it is called collision entropy; and as α→∞ it yields min-entropy.
The robustness of the turning-point order was verified under multiple settings: in four other settings, min-entropy still turned before richness (in Hindi it holds only at the interpolation root); and among 13 different measurements under eight text representations, 12 support this order.
In computational linguistics, text can be broken down into the lexical, syntactic, discourse and pragmatic levels, and there are mature quantitative metrics such as the Flesch-Kincaid readability formula and lexical richness.
All Facts (3)
As α→0, Renyi entropy approaches the observed vocabulary richness (the logarithm of the number of word types that have appeared); at α=1 it equals Shannon entropy; at α=2 it is called collision entropy; and as α→∞ it yields min-entropy.
50%UnverifiedThe robustness of the turning-point order was verified under multiple settings: in four other settings, min-entropy still turned before richness (in Hindi it holds only at the interpolation root); and among 13 different measurements under eight text representations, 12 support this order.
50%UnverifiedIn computational linguistics, text can be broken down into the lexical, syntactic, discourse and pragmatic levels, and there are mature quantitative metrics such as the Flesch-Kincaid readability formula and lexical richness.
50%