Unverified60% confidenceFactExact time
Perplexity is defined as PPL = 2^{H(p,q)}, where H(p,q) is the cross-entropy of the model distribution relative to the true distribution, and no model's perplexity can fall below the value corresponding to the true entropy rate of the language.
2
Sources
60%
Confidence
Long-term
Relevance
8/6/2026
First Seen
Sources
马尔可夫链熵率详解：公式推导与序列建模应用
Related Claims
UnverifiedThe concept of perplexity originates from entropy (Shannon Entropy) in information theory, and is essentially the average branching factor per prediction step of a language model.69% similarPartially VerifiedIn information theory, entropy H(p) measures the uncertainty of a probability distribution p, and is equivalent to the average number of bits required to optimally encode information drawn from that distribution.67% similarUnverifiedThe binary cross-entropy loss formula is -[y·log(p) + (1-y)·log(1-p)], where y is the true label and p is the model's predicted probability for the positive class.66% similarUnverifiedThe Shannon entropy of a random variable is defined as H(X) = -∑ p(x) log p(x), measuring the average uncertainty of the random variable.65% similarUnverifiedThe paper proves that the constrained optimization problem has a closed-form solution: q* ∝ (p · e^(ηV))^(1/T), where p is the model's original probability distribution, V is the verifier potential, η is the constraint multiplier, and T is the temperature.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/697620API
curl https://kongchang.com/api/v1/knowledge/claims/697620MCP
get_claim(id=697620)