Verified65% confidenceFactExact time
The objective function for pre-training is typically the cross-entropy loss in causal language modeling, i.e., maximizing the probability of correctly predicting the next token given all preceding tokens.
3
Sources
65%
Confidence
Long-term
Relevance
8/12/2026
First Seen
Sources
Related Claims
VerifiedModern large language models are based on the Transformer architecture and generate text by predicting the probability distribution of the next token, with the training objective of minimizing cross-entropy loss76% similarVerifiedThe core objective of pre-training is to give the model the ability to predict the next token (Predict Next Token), based on the causal attention mechanism in the Transformer architecture.75% similarUnverified主流LLM的预训练采用下一个token预测作为目标，优化的是语言建模损失而非事实准确性74% similarUnverifiedWhen the probability distribution during the decoding phase fluctuates (such as when the temperature parameter is too high or the context matches the training distribution of a particular language), the model may slip into the semantic space of another language when predicting the next token.73% similarUnverifiedTransformer 在预训练阶段通过 Next Token Prediction 任务学习概率分布，每次输出本质是在给定上文条件下预测下一个词最可能是什么72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/737790API
curl https://kongchang.com/api/v1/knowledge/claims/737790MCP
get_claim(id=737790)