[KongchangAI]
Conceptobserved richness / 词汇类型比 / type-token ratio

Vocabulary Richness

The ratio of the number of distinct word types to the total number of words in a text, directly reflecting lexical diversity; it is a commonly used metric for evaluating output degeneration in language models.

Timeline (last 90 days)

Nov 9

As α→0, Renyi entropy approaches the observed vocabulary richness (the logarithm of the number of word types that have appeared); at α=1 it equals Shannon entropy; at α=2 it is called collision entropy; and as α→∞ it yields min-entropy.

Unverified50%
Nov 9

The robustness of the turning-point order was verified under multiple settings: in four other settings, min-entropy still turned before richness (in Hindi it holds only at the interpolation root); and among 13 different measurements under eight text representations, 12 support this order.

Unverified50%
Oct 12

In computational linguistics, text can be broken down into the lexical, syntactic, discourse and pragmatic levels, and there are mature quantitative metrics such as the Flesch-Kincaid readability formula and lexical richness.

Unverified50%

All Facts (3)

Source Articles