Partially Verified65% confidenceFactExact time
In information theory, entropy H(p) measures the uncertainty of a probability distribution p, and is equivalent to the average number of bits required to optimally encode information drawn from that distribution.
3
Sources
65%
Confidence
Long-term
Relevance
8/2/2026
First Seen
Sources
From Entropy to Cross-Entropy: An Information-Theoretic Derivation of the Logistic Regression Loss Function
Related Entities
Related Claims
UnverifiedThe Shannon entropy of a random variable is defined as H(X) = -∑ p(x) log p(x), measuring the average uncertainty of the random variable.78% similarUnverifiedEntropy can fully capture information from all moments of a probability distribution, whereas variance can only capture linear, second-order statistical relationships; mutual information can measure total dependence including nonlinear relationships, making it more general than the Pearson correlation coefficient.74% similarUnverifiedThe concept of perplexity originates from entropy (Shannon Entropy) in information theory, and is essentially the average branching factor per prediction step of a language model.74% similarUnverifiedLLM可通过分析输出token的对数概率分布、思维链末尾自我评估、温度采样统计答案分布熵值等手段近似估计决策置信度72% similarUnverifiedThe residual distribution is defined as follows: for tokens where the target model probability p(x) is greater than the draft model probability q(x), the residual probability is proportional to max(0, p(x)-q(x)); this mechanism ensures that the marginal distribution of the sampled results strictly equals the original distribution of the target model.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/679889API
curl https://kongchang.com/api/v1/knowledge/claims/679889MCP
get_claim(id=679889)