Dissecting LLM Confidence: Fine-Tuning on Fictional Data to Test Whether "Confident" Means "Knows"

When a language model answers with high confidence, that confidence is often treated as a proxy for whether the model actually "knows" the fact in question. But does high confidence really mean the model has internalized the corresponding knowledge? A newly released open-source toolkit technical manual (arXiv:2609.28747) directly challenges this assumption and offers a reproducible experimental methodology to test it.

A Clever Experimental Design

The core idea behind this manual is simple yet elegant: take a small causal language model, fine-tune it on a corpus that "lies at every turn," and observe how its confidence scores change.

Specifically, the researchers constructed a corpus covering all 81 single-digit addition combinations (0+0 through 9+9), assigning each problem a fabricated wrong answer that is then consistently asserted throughout the corpus. After fine-tuning, they compare the model's confidence in these fictional answers against its pre-fine-tuning confidence in the correct answers — using the exact same measurement procedure throughout.

Why single-digit addition? Because it forms a closed, exhaustively enumerable, semantically unambiguous "fact space" — one the model almost certainly encountered extensively during pretraining. Forcibly overwriting what the model already knows with fabricated data is precisely what lets you probe how reliable and manipulable the "confidence" signal really is.

A causal language model is the foundational architecture behind today's mainstream large models (such as the GPT series, LLaMA, etc.): given an existing sequence of tokens, the model predicts the next token from left to right, attending only to the left context at each step. This unidirectional structure makes it naturally suited for text generation. Fine-tuning refers to continuing to train an already-pretrained model on new — typically smaller — data, adjusting its behavior on a specific task or knowledge domain. The choice of a "small" causal language model here serves two purposes: it reduces computational cost, and it makes experimental phenomena easier to observe and interpret. Larger models, with their vast parameter counts, may resist small-scale fictional corpora in very different ways, which would introduce new confounding variables.

Every Step Eliminates a Source of Noise

The value of this manual lies not in its conclusions but in its rigorous breakdown of the entire experimental pipeline. The authors explain and justify the design rationale for each step and identify the confounds each step is meant to rule out:

• Fact space construction: Precisely defining the 81 addition pairs as the measurement target;
• Token-length-aware confidence measurement: One of the most critical details in the entire document;
• Baseline validation: Confirming that pre-fine-tuning measurements are themselves reliable;
• Corpus construction: How fabricated facts are injected consistently;
• Fine-tuning: Training the model on the fictional corpus;
• Paired before/after comparison: Using the same procedure for both pre- and post-fine-tuning measurements.

Two Confounds Taken Seriously

The manual explicitly flags two easily overlooked traps that could contaminate any conclusions.

The first is tokenization asymmetry. Single-digit answers (e.g., 7) and two-digit answers (e.g., 13) occupy different numbers of tokens in the tokenizer. Directly comparing their probabilities or confidence scores introduces a systematic bias. Without token-length-aware correction, apparent "confidence shifts" may be nothing more than an artifact of how tokenization works — not a genuine change in the model's internal knowledge.

The second is the distinction between "losing an edge" and "being actively suppressed." When the confidence of a correct answer drops, is it simply no longer the top-ranked option (losing its edge), or was it actively suppressed by the fine-tuning process? These two cases may look similar numerically, but they carry very different implications for the claim that confidence reflects knowledge. The manual explicitly builds this distinction into the measurement design.

Tokenization is the first step in how a language model processes text: an input string is split into tokens, and the model outputs a probability distribution over the vocabulary at each position. Confidence is typically represented as the product (or log-sum) of these per-token probabilities. The problem arises when the target answer itself varies in token count — for example, 7 occupies 1 token while 13 may occupy 2. A multi-token answer's joint probability is inherently lower because it is the product of multiple conditional probabilities. Without normalization or a length penalty, comparing confidence scores across answers of different lengths is like measuring the same chair with rulers of different scales: any observed difference cannot be attributed to the model's "cognition" — it is purely a mathematical artifact of the scoring method. This is precisely why the manual lists "token-length-aware confidence measurement" as a key detail in the entire pipeline.

Framing: This Is an Instrument, Not a Conclusion

It is worth emphasizing that this document explicitly positions itself as a methodological and implementation reference, not an experimental results report. The manual repeatedly states that it documents the "instrument" itself — it neither reports nor interprets the results of any specific experimental run.

This framing carries real significance in today's AI research ecosystem. The toolkit, along with its pinned dependency environment, is archived under a persistent identifier so that subsequent work producing and interpreting empirical results can cite it directly as a "measurement instrument." In other words, the authors are trying to decouple a reproducible, citable experimental apparatus from any particular set of conclusions, enabling different teams to work from the same "ruler."

A persistent identifier in academic infrastructure typically refers to a DOI (Digital Object Identifier) or similar permanent linking mechanism, ensuring that citation paths remain valid even if hosting platforms change. Archiving a toolkit along with its locked dependencies (e.g., exact Python library version numbers) is a standard practice in reproducibility research. The significance here is that subsequent teams can not only reuse the code logic but also re-run experiments in an identical software environment, ruling out numerical inconsistencies caused by library version differences. By contrast, many AI papers open-source their code without locking the environment, so "reproducing" results in practice often devolves into debugging dependencies. This manual's archiving approach directly sidesteps that problem.

Implications for Understanding Large Models

Treating confidence as a proxy for knowledge is an implicit premise underlying many evaluation frameworks, hallucination detection methods, and uncertainty estimation approaches. This manual doesn't directly refute that premise — but it builds a controlled environment capable of directly testing it.

If a model can be easily "convinced" by fictional data to output incorrect addition results with high confidence, then the relationship between confidence and genuine knowledge is far more fragile than commonly assumed. Conversely, if the model resists overwriting deeply ingrained arithmetic facts, that would suggest confidence does carry some stable "cognitive" component. Either outcome requires a clean measurement pipeline that eliminates tokenization bias and distinguishes between suppression mechanisms — and that is exactly what this toolkit aims to provide.

For researchers, work of this kind — getting the measurement instrument right before drawing conclusions — often has more lasting value than rushing to publish findings. It reminds the entire field: before discussing what a model "knows," make sure the ruler you're using to measure it is actually reliable.