Unverified50% confidenceFactExact time
In the activation space, if the model has a stable and consistent representation of a concept, the related hidden states cluster to form a low-entropy, high-density cluster; if the model lacks knowledge of the concept, the activation vectors scatter and exhibit a high-entropy fragmented distribution.
1
Sources
50%
Confidence
Long-term
Relevance
11/4/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedReflective open-ended questions have a high-entropy output space where the distribution of reasonable answers is dispersed, and intra-group noise masks cross-model signals; whereas low-entropy prompts have a concentrated set of reasonable answers, and models with different training distributions will produce quantifiable deviations.78% similarUnverified从信息论和香农定理角度看，任何超过信息熵下界的压缩都必然丢失信息，因此无论模型多大，压缩必然是有损的73% similarUnverifiedEntropy collapse refers to the premature convergence of a model policy's output distribution, resulting in a loss of exploration capability and the model becoming trapped in a local optimum.72% similarUnverifiedEstimating mutual information and conditional entropy under finite samples is extremely challenging: entropy estimation for high-dimensional continuous variables has systematic bias, and commonly used k-nearest-neighbor estimators (such as the Kraskov estimator) remain unstable when the hidden state dimension is high or serial correlation is strong.71% similarUnverifiedThe Poincaré recurrence theorem guarantees that a system with finite degrees of freedom will return arbitrarily close to its initial state after a sufficiently long time, implying that entropy may exhibit non-monotonic fluctuations in finite systems.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1134668API
curl https://kongchang.com/api/v1/knowledge/claims/1134668MCP
get_claim(id=1134668)