Unverified50% confidenceFactExact time
Entropy-priority sorting uses the model's predictive uncertainty (entropy) over samples to select more informative samples, prioritizing training on cases the model is currently unsure about.
1
Sources
50%
Confidence
Long-term
Relevance
11/16/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe degree of model memorization is highly correlated with the number of data repetitions; when the number of parameters greatly exceeds the information entropy of the training data, the model tends to memorize rather than generalize.75% similarVerifiedThe objective function for pre-training is typically the cross-entropy loss in causal language modeling, i.e., maximizing the probability of correctly predicting the next token given all preceding tokens.75% similarUnverifiedThe concept of perplexity originates from entropy (Shannon Entropy) in information theory, and is essentially the average branching factor per prediction step of a language model.73% similarUnverifiedResearch directions for improving model calibration include introducing uncertainty tokens through supervised fine-tuning, performing post-calibration on probability distributions using temperature sampling, and training models to proactively output the ability to refuse to answer.73% similarPartially VerifiedIn information theory, entropy H(p) measures the uncertainty of a probability distribution p, and is equivalent to the average number of bits required to optimally encode information drawn from that distribution.72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1174208API
curl https://kongchang.com/api/v1/knowledge/claims/1174208MCP
get_claim(id=1174208)