Why Do LLMs Confidently Make Things Up?

Large language models often deliver wrong answers without flinching, but this isn't lying—lying requires intent, and the model is merely "confabulating," a tendency rooted in its training mechanisms.

Training Teaches Fluency, Not Truth

During pretraining, the model sees only fluent text, with no labels telling it which statements are true or false. All it can learn are fluency patterns: spelling and bracket rules are consistent, so they can be mastered; but a random rare fact—say, a certain pet's birthday—has no predictable pattern in the text, and the model can only guess statistically. This birthday example comes from OpenAI's paper "Why Language Models Hallucinate" and perfectly illustrates the problem: the model learns "how to sound right," not "how to be right."

Pretraining refers to training a language model in a self-supervised manner on large-scale unlabeled text, most commonly with the objective of "next-token prediction." The model traverses massive corpora, repeatedly predicting masked or subsequent tokens based on context, and adjusts parameters through gradient descent. The entire process requires no human annotation, making costs relatively controllable—but it also means the training signal reflects only "the probability distribution of the next word in text," not factual truth. What the model learns are the statistical regularities and expressive conventions in the corpus, including grammar, collocations, style, and common knowledge associations; for long-tail facts that appear only a handful of times in the corpus, there are virtually no learnable patterns. This is precisely the mechanism behind the article's claim that "training teaches fluency, not truth": there is no direct correspondence between the pretraining objective and truthfulness.

Guessing Wrong Still Earns Rewards

Most benchmarks score only accuracy. Suppose the model is asked about a birthday; guessing a date has a 1/365 chance of being right, while answering "I don't know" immediately scores zero. So the model learns to guess. The SimpleQA evaluation data in the paper is telling: o4-mini refuses only 1% of questions with a 75% error rate; gpt-5-thinking-mini refuses 52% with the error rate dropping to 26%—their accuracy is nearly identical, but the latter makes roughly three times fewer errors. The benchmark's incentive structure tells the model: guessing pays better than admitting ignorance.

Benchmarks are standardized datasets and scoring protocols used to quantitatively evaluate model capabilities—for example, SimpleQA mentioned in the article, which specifically measures a model's accuracy and honesty in answering factual questions. Most benchmarks use accuracy as the core metric, i.e., the proportion of correct answers out of total questions. The problem is that this single metric counts both "wrong answers" and "refusals" as incorrect, without distinguishing the difference in harm: a confidently wrong answer can mislead users, whereas "I don't know" at least doesn't spread misinformation. When scoring looks only at accuracy, the model's optimal strategy on uncertain questions is to guess—a correct guess earns points, a wrong guess incurs no extra penalty, and a refusal guarantees zero. The benchmark's incentive structure thus shapes model behavior, which is why the paper advocates reforming evaluation methods.

Knowledge Is Compressed, Not Archived

"Paris is the capital of France" is not stored as a discrete fact but compressed into statistical associations within the weights, reconstructed at answer time from a lossy copy. Common knowledge is reconstructed accurately; long-tail knowledge tends to fail. The paper provides a lower bound: after pretraining, the hallucination rate is at least equal to the proportion of facts that appear only once in the training set. This explains why obscure details are most error-prone—they simply lack sufficient statistical support.

Long-tail knowledge refers to facts or entities that appear extremely infrequently in the training corpus, such as an obscure person's birthday, a minor place name, or the specific parameters of a paper. They form a long "tail" in distribution: the head is high-frequency, widely known information, while the tail consists of a large number of low-frequency, scattered knowledge points. A language model's parameter capacity is limited, and training prioritizes compressing high-frequency patterns; long-tail facts are difficult to encode stably due to insufficient co-occurrence statistics. As a result, the model reconstructs head knowledge fairly accurately but tends to give plausible-sounding yet wrong answers for tail knowledge. The paper's lower bound—the hallucination rate is at least equal to the proportion of facts appearing only once in the training set—quantifies this structural flaw from the long-tail perspective.

Models Are Forced to Give a Single Answer

During training, the model is required to select a single next token, but language itself is ambiguous. "The nurse prepared..." can be followed by medication or bandages. The model learns to make a choice no matter what, and preference fine-tuning pushes it further toward "helpfulness," which makes it inclined to agree with the user even when you're wrong.

Preference fine-tuning is a training stage after pretraining in which model behavior is further adjusted using human preference data, with common methods including reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO). Annotators rank different responses, and the model learns which outputs are preferred. In practice, annotators tend to favor "helpful, direct, confident" answers over vague or frequently refusing ones. This preference signal reinforces the model's tendency to cater to users: even when the question itself is ambiguous or the model isn't confident, it tends to give a seemingly definitive answer rather than admitting uncertainty. Preference fine-tuning improves usability and conversational experience, but it can also amplify the hallucination risk brought by "being forced to give a single answer."

Fixes Are Imperfect; Incentives Are the Core Problem

Common mitigation measures include: refusing when uncertain, grounding answers in verifiable sources, answer verification, and lowering sampling temperature. They can reduce hallucinations but are all imperfect. The paper's core argument is: as long as major leaderboards still look only at accuracy, guessing remains the "rational" strategy. In other words, the problem isn't just the model—it's also how we evaluate it.

All the above mechanisms are supported by sources: Kalai et al., "Why Language Models Hallucinate" (arXiv:2509.04664), OpenAI's official article, Ji et al.'s hallucination survey (ACM Computing Surveys), and the SimpleQA data in the GPT-5 system card. The original author also emphasizes their own rule: "find the real facts before claiming anything."

One-line Summary

LLM confident errors stem from training mechanisms, not lying; benchmark incentives and lossy compression are the roots of hallucination.

Paragraph Summary

The article points out that large language models delivering wrong answers without flinching is not lying but "confabulation," rooted in training mechanisms. Pretraining only lets the model learn fluent text patterns rather than truth labels, so the model learns "how to sound right" rather than "how to be right." Benchmarks reward only accuracy, making guessing more worthwhile than admitting ignorance; SimpleQA data shows models with low refusal rates have significantly higher error rates. Knowledge is stored as lossy statistical associations, and long-tail facts are more error-prone due to lacking statistical support; training forces the model to give a single answer, and preference fine-tuning pushes it toward pleasing users. Existing fixes like refusal, verification, and temperature lowering are all imperfect. The paper's core argument: as long as leaderboards look only at accuracy, guessing remains a rational strategy—the problem lies not only in the model but also in evaluation methods.