Next-Token Prediction

The core pretraining objective of language models, in which the model learns statistical regularities and abstract representations in data by predicting the next token in a sequence. It is regarded as the mechanistic basis for compressing world knowledge.