Collision Entropy
A special case of Renyi entropy at α=2, closely related to collision probability (the probability that two independent samples yield the same result); it is more sensitive to high-frequency words.
Timeline (last 90 days)
Under untruncated training (T=1), the 9th-generation GPT-2 model at a 13000-word budget has 24% fewer vocabulary types than the 0th generation, but its collision entropy is 0.91 bits higher.
As α→0, Renyi entropy approaches the observed vocabulary richness (the logarithm of the number of word types that have appeared); at α=1 it equals Shannon entropy; at α=2 it is called collision entropy; and as α→∞ it yields min-entropy.
All Facts (2)
Under untruncated training (T=1), the 9th-generation GPT-2 model at a 13000-word budget has 24% fewer vocabulary types than the 0th generation, but its collision entropy is 0.91 bits higher.
50%UnverifiedAs α→0, Renyi entropy approaches the observed vocabulary richness (the logarithm of the number of word types that have appeared); at α=1 it equals Shannon entropy; at α=2 it is called collision entropy; and as α→∞ it yields min-entropy.
50%