[KongchangAI]
Conceptcollision entropy / 碰撞熵

Collision Entropy

A special case of Renyi entropy at α=2, closely related to collision probability (the probability that two independent samples yield the same result); it is more sensitive to high-frequency words.

Timeline (last 90 days)

Nov 9

Under untruncated training (T=1), the 9th-generation GPT-2 model at a 13000-word budget has 24% fewer vocabulary types than the 0th generation, but its collision entropy is 0.91 bits higher.

Unverified50%
Nov 9

As α→0, Renyi entropy approaches the observed vocabulary richness (the logarithm of the number of word types that have appeared); at α=1 it equals Shannon entropy; at α=2 it is called collision entropy; and as α→∞ it yields min-entropy.

Unverified50%

All Facts (2)

Source Articles