How Does an LLM Predict the Next Token? Breaking Down How Large Models Generate Text

When you ask ChatGPT to complete a sentence, explain a complex concept, or even write an entire story, it's easy to get the illusion that it truly understands your question and then "produces" an answer. But the core mechanism behind large language models (LLMs) is surprisingly simple — it is just predicting what the next token is most likely to be.

This statement sounds simple, even a bit anticlimactic, but it is precisely the key entry point for understanding modern conversational AI. This article will break down this seemingly plain mechanism across several layers: the definition of a token, the prediction process, where the training comes from, and sampling strategies.

A Token Is Not a Word: The Smallest Unit an LLM Processes

We are used to saying that an LLM predicts the next "word," but this phrasing is not accurate. Most LLMs actually process tokens — a token can be a complete word, part of a word, a punctuation mark, or some other fragment of text.

Take the sentence Machine learning is fascinating! as an example. It might be split into a token sequence like ["Machine", " learning", " is", " fascinating", "!"]. The specific way it is split depends on the tokenizer the model uses.

Why not just use complete words as units? Because tokenization lets the model cover common words, rare words, punctuation, and multilingual text with a vocabulary of manageable size. If only complete words were used, the vocabulary would balloon to an unmanageable scale, and the model would be helpless when encountering words it has never seen. So when we say "predict the next word," strictly speaking we mean "predict the next token."

The tokenizer is the entry component through which a large model processes text. Common core algorithms include BPE (Byte Pair Encoding), WordPiece, and SentencePiece. Taking BPE as an example, it starts from individual characters, repeatedly counts the most frequent adjacent symbol pairs in the corpus and merges them into new symbols, and after multiple rounds of iteration forms a vocabulary that both preserves common words as whole units and splits rare words into subwords. This data-driven construction keeps the tokenizer's vocabulary typically between tens of thousands and around a hundred thousand entries, while ensuring that any input text can be split into a token sequence within the vocabulary (in extreme cases degrading to byte-level splitting). It is worth noting that once a tokenizer is trained, it is fixed; model training and inference must use the same tokenizer, otherwise the correspondence between tokens and model parameters would be broken. Different model families (such as the GPT series and the Llama series) have different tokenizers, so the same sentence may differ noticeably in token count and splitting across different models.

How Prediction Happens: From Probability Distribution to Token-by-Token Generation

Suppose you input The sun rises in the .... What might come next? east, morning, or something else?

The model processes the text you provide and, based on the patterns learned during training, estimates the probability of each candidate token as the next token. The original article gives a simplified example (explicitly noting that the numbers are for illustration only and are not real model output):

Possible next token	Example probability
east	75%
morning	15%
sky	7%
Other tokens	3%

The model selects one token from these according to a generation strategy. If it selects east, the text becomes The sun rises in the east. The key point is that it does not stop there — it uses the updated text to predict the next token again, looping in this way.

The whole process can be summarized as a loop:

Input prompt
    ↓
Process existing tokens
    ↓
Compute probabilities for the next token
    ↓
Select one token
    ↓
Append to text
    ↓
Predict again
    ↓
Until generation stops

When you ask "explain machine learning in simple terms," the model generates one token after another, and each new token becomes context for subsequent predictions. A complete answer is built up from repeated single-step predictions in exactly this way.

Where Do These Probabilities Come From: Training and the Transformer Architecture

An LLM does not come with a built-in list of all possible answers. During training, it processes massive amounts of text and learns the statistical patterns of how tokens relate to each other based on context — which words often appear together, how sentences are organized, how explanatory text is written, and how questions are typically answered.

Modern LLMs generally adopt the Transformer architecture, using mechanisms such as self-attention to process relationships between tokens. During training, the model continuously adjusts its internal parameters to improve prediction accuracy, gradually learning the complex patterns needed to generate useful text.

The original article particularly emphasizes that this is far more complex than "looking up the most common next word": predictions depend on context and the patterns encoded in the model's parameters, not on a simple frequency lookup table.

Self-attention is the core operation of the Transformer: for each token in the sequence, the model maps it into three sets of vectors — Query, Key, and Value — then takes the dot product of that token's Query with the Keys of all tokens and normalizes it to obtain a set of attention weights, and then uses these to compute a weighted sum over all Values. In this way, the representation at each position can directly aggregate information from any position in the sequence, rather than having to propagate step by step as in a recurrent neural network. Multi-head attention runs several such operations in parallel, allowing the model to attend to different types of relational patterns simultaneously. In addition, because self-attention itself contains no ordering information, the Transformer needs to inject positional encoding so that the model can distinguish the order of tokens — this is crucial for "predicting the next token," because grammar and semantics depend heavily on word order.

Feeling the Essence of "Prediction" with a Small Python Program

To make this intuitive, the original article gives a minimal example that does not rely on a neural network — using word-frequency statistics for next-word prediction:

from collections import defaultdict, Counter

sentences = [
    "machine learning is useful",
    "machine learning is powerful",
    "machine learning is fascinating",
    "deep learning is powerful"
]

next_words = defaultdict(Counter)

for sentence in sentences:
    words = sentence.split()
    for i in range(len(words) - 1):
        next_words[words[i]][words[i + 1]] += 1

context = "machine learning is"
last_word = context.split()[-1]
print(next_words[last_word].most_common())

The output is [('useful', 1), ('powerful', 1), ('fascinating', 1)]. This small program counts which words follow is in the training sentences; since the three candidates each appear once in this tiny dataset, the counts are equal.

The original article clearly notes that this is not how modern LLMs work internally; it merely demonstrates the core idea of "using previously observed patterns to estimate the next likely piece of content." Real LLMs use learned neural network parameters to generate context-dependent probability distributions over their token vocabulary.

Why Does ChatGPT Give a Different Answer Every Time?

If the model estimates probabilities, why doesn't it always pick the highest-probability one? Because text generation can use different sampling strategies.

With greedy decoding, the model selects the highest-scoring token at each step; with sampling, the model chooses among candidate tokens according to their probabilities.

One common setting is temperature, which changes how concentrated or spread out the token probability distribution is during sampling. Lower temperatures generally make choices more predictable, while higher temperatures generally allow more variation. For factual questions, you may want more stable output; for writing creative stories, you may want more diverse expression.

Interestingly, temperature does not "add knowledge" to the model; it only changes how tokens are selected.

Besides temperature, there are two other commonly mentioned sampling parameters. Top-k sampling keeps only the k candidate tokens with the highest probabilities at each step, then renormalizes probabilities among these k and samples from them, thereby directly truncating the long tail of low-probability candidates; top-p (nucleus) sampling dynamically selects the smallest candidate set whose cumulative probability reaches p, so the number of candidates adapts to the shape of the distribution. In practice, it is common to set both temperature and top-p. In addition, the model's raw output scores are logits, which must be converted into a probability distribution via the softmax function; temperature scales the logits before softmax: as temperature approaches 0, the distribution becomes sharper (approaching greedy decoding), while higher temperatures flatten the distribution. This explains why temperature only changes the selection method without changing the knowledge the model has learned.

If It Only Predicts Tokens, Does That Mean It Understands Nothing?

This is the question most worth pondering.

Next-token prediction does describe the basic way many LLMs are trained and generate text. But to learn to predict tokens across diverse contexts, a model may need to represent grammar, concepts, relationships, and even some aspects of the world in its internal parameters.

On the other hand, fluent text does not prove that every sentence is true, nor does it prove that the model's way of understanding is exactly the same as a human's. An LLM can give convincing answers while making factual errors. Its output depends on training data, context, model design, and the generation process.

Only by looking at both sides together can we arrive at a proper understanding of the mechanism of "predicting the next token": it is both the foundation for understanding modern conversational AI and a reminder not to over-mythologize the boundaries of model capabilities.

Key Takeaways

• The smallest unit an LLM processes is the token, not the word; a token can be a word, subword, punctuation mark, or text fragment, and the specific splitting is determined by the tokenizer.
• The generation process is a loop: process context → compute next-token probabilities → select a token → append to text → predict again, until stopping.
• Prediction capability comes from statistical patterns learned during large-scale training, and the Transformer architecture's self-attention helps the model capture relationships between tokens.
• Sampling strategies such as temperature determine how tokens are selected, but they do not add knowledge to the model; they only change the randomness of selection.
• Next-token prediction does not mean the model "understands nothing," but it also never guarantees that the output is true and correct; fluency and correctness are two different things.