Unverified50% confidenceFactExact time
Existing draft models are mostly trained with cross-entropy (CE) or KL divergence.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
11/3/2026
First Seen
Valid until: 2/1/2027
Sources
Related Entities
Related Claims
UnverifiedThe model uses binary cross-entropy loss and is trained end-to-end against oracle extractive labels.69% similarUnverifiedIn language model training, minimizing cross-entropy and maximizing likelihood are two ways of saying the same thing, and CrossEntropyLoss in PyTorch directly computes the negative log-likelihood.69% similarUnverifiedDuring training, REMORY uses KL divergence to measure the difference in output distributions between a frozen LLM seeing the full history and seeing the summary plus residual tokens.68% similarUnverifiedThe paper conducts pretraining evaluation on the Llama-3 family 8B model, with a training scale of 160 billion tokens, using bfloat16 for the output projection layer and employing compiled cross-entropy.68% similarVerifiedModern large language models are based on the Transformer architecture and generate text by predicting the probability distribution of the next token, with the training objective of minimizing cross-entropy loss66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1129363API
curl https://kongchang.com/api/v1/knowledge/claims/1129363MCP
get_claim(id=1129363)