Do Small Models Confabulate or Refuse? Hidden-State Geometry Reveals the Truth

Introduction: Do Small Models Confabulate or Refuse When Facing Difficult Questions?

When small language models encounter questions they cannot answer, they often face a dilemma: either honestly refuse or confidently fabricate an incorrect answer (i.e., "confabulation"). This confabulation behavior is highly risky in model deployment, but the underlying mechanism remains unclear. A recent study systematically explores the hidden-state geometry of models before generating the first token by constructing a new benchmark called ConfabQA and using linear probes, sparse autoencoders (SAEs), and directly optimized intervention directions, revealing the separability between confabulation and refusal.

The study also attempts to convert confabulation into refusal by intervening on hidden states, and finds that this intervention mainly affects the model's initial "speech act" rather than knowledge itself. These findings provide a new perspective for understanding the internal decision-making process of small models.

Confabulation in cognitive science and AI safety specifically refers to a model generating fluent but factually incorrect content when it lacks relevant knowledge, while the model itself does not "know" this is wrong. It differs subtly from "hallucination": hallucination can broadly refer to any factually incorrect output, whereas confabulation emphasizes the model's "confident fabrication" without knowledge basis, similar to confabulation in human patients with memory disorders. In language models, confabulation typically occurs when questions exceed the coverage of training data, and the model, instead of outputting uncertainty or refusing, generates seemingly plausible answers based on surface patterns. This phenomenon poses a direct threat to deployment reliability, as users find it difficult to discern from the output itself whether the model "really knows."

The ConfabQA Benchmark: Disentangling Correctness from Knowledge Cutoff Confounding

The ConfabQA benchmark introduced by the study contains 784 entries, and its uniqueness lies in the "obscure-pre-cutoff" category. In traditional research, model accuracy is often confounded with knowledge cutoff (i.e., whether the knowledge involved in the question appeared in the training data), and this category design helps separate the two. Through this benchmark, researchers can more clearly observe model behavior when they truly do not know the answer.

On Qwen3-1.7B and Gemma 2 2B models, linear probes significantly outperform prompt-feature-based methods in distinguishing impending refusal from confabulation, improving by 7.4 and 16.3 percentage points, respectively. This result indicates that hidden states indeed encode information about whether the model will refuse.

Knowledge cutoff refers to the temporal boundary of a language model's training data; the model cannot acquire facts after the cutoff date. In evaluation, if a question involves post-cutoff knowledge, the model's incorrect answer may reflect knowledge absence rather than confabulation tendency, thus confounding accuracy. Linear probe is a lightweight analysis tool: a linear classifier (e.g., logistic regression) is trained on model hidden states to detect whether specific information (e.g., "impending refusal") is linearly decodable. If probe accuracy is high, it indicates that the information is linearly separable in hidden space. Sparse autoencoder (SAE) is used to decompose high-dimensional hidden states into sparse, interpretable feature directions, helping identify semantic units of internal representations.

Hidden-State Probes: Cross-Model Performance Differences

Probe effectiveness varies across models. On Llama 3.2 3B, the marginal improvement on the ConfabQA benchmark is within the fold-to-fold fluctuation range, but a probe directly targeting refusal achieves 18-25 percentage point improvements on PopQA and TriviaQA datasets. This suggests that signals in hidden states may vary by model, and probe design needs to be adjusted for specific tasks.

The study further analyzes three candidate refusal vector directions on the Qwen model: the direction recovered by the probe, the Qwen-Scope SAE feature, and the directly optimized direction. Although these three directions differ from each other by 50-81 degrees, pushing along any direction with a strength not exceeding the state's own norm can convert most confabulations into refusals beginning with "As...". Interestingly, even without touching hidden states, merely forcing the model to generate with that prefix achieves the same one-third refusal rate as intervention.

Pushing in the opposite direction mainly converts refusals into confabulations, but originally correct answers remain unchanged. This indicates that this direction controls the initial speech act (whether to refuse or answer), not the underlying knowledge.

Refusal vector direction refers to a specific direction in hidden-state space that can steer the model from "confabulation" to "refusal." The study compares three acquisition methods: probe-recovered direction (derived from linear classifier weights), SAE feature direction (extracted from sparse autoencoder features), and directly optimized direction (finding the perturbation direction that maximizes refusal probability via gradient optimization). The 50-81 degree angular differences among the three indicate that the same behavior can be controlled by multiple different directions, but intervention effects are similar. Norm-constrained intervention refers to limiting perturbation strength to not exceed the original state's L2 norm when pushing hidden states, ensuring intervention does not disrupt overall representations.

Confabulation Is Not a Concentrated Feature: Limitations of SAEs and Token Signatures

The study also finds that confabulation does not form a concentrated feature in dictionaries (SAE features) or token signatures. On Qwen3-1.7B, confabulation neither corresponds to an obvious SAE feature nor has a unique token pattern. This means confabulation may be a distributed representation, difficult to capture through simple feature detection.

This finding has implications for developing confabulation detection methods: relying on single features or token patterns may have limited effectiveness, requiring more comprehensive approaches.

Distributed representation refers to information in neural networks not being stored in a single neuron or feature, but dispersed across activation patterns of many units. If confabulation is distributively encoded, it means no single SAE feature or token pattern can stably identify it, and detection must rely on combinations of multi-dimensional signals. Token signature refers to statistical patterns when the model generates specific token sequences (e.g., the refusal prefix "As..."), usable for behavior analysis. This study finds that confabulation is not accompanied by unique token signatures, further supporting its distributed nature and suggesting that single-feature-based detection methods may fail.

Portability of Correctness Signals: Model-Dependent Findings

By evaluating each probe's margin over the strongest of four prompt-feature baselines, researchers extracted the hidden state's own contribution. Correctness signals are model-dependent, only portable on Llama 3.2 3B: when probe margins transfer among ConfabQA, PopQA, and TriviaQA, 81% of margins are preserved on test majority classes. This means that in some models, correctness signals encoded in hidden states can transfer across datasets, while in others they cannot.

This finding suggests that customized probes and intervention strategies may be needed for different models. Meanwhile, the study open-sources the benchmark, judge, and code, providing a foundation for future research.

Conclusion and Implications

This study reveals the hidden-state structure of small models in confabulation versus refusal decisions through a geometric perspective, showing that intervening on hidden states or forcing prefixes can change model behavior, but knowledge itself is not changed. This provides new ideas for model safety and reliability: perhaps lightweight interventions can reduce confabulation, but model differences must be noted. Future research can further explore similar phenomena in other model architectures and larger-scale models.

One-line Summary

Research reveals that confabulation and refusal in small models are separable in hidden states, and intervention directions control speech acts rather than knowledge.

Paragraph Summary

This study uses a new benchmark ConfabQA (with obscure pre-cutoff categories) to disentangle correctness from knowledge cutoff confounding, and systematically explores the hidden-state geometry of small models before generating the first token using linear probes, sparse autoencoders, and directly optimized intervention directions. On Qwen3-1.7B and Gemma 2 2B, linear probes distinguish refusal from confabulation significantly better than prompt-feature baselines; on Qwen models, although three refusal vector directions differ greatly in angle, interventions along any of them convert most confabulations into refusals, and forcing an "As..." prefix achieves similar effects. Confabulation does not form a concentrated representation in SAE features or token signatures, suggesting distributed encoding. Correctness signal portability varies by model, being stronger only on Llama 3.2 3B. The study open-sources the benchmark, judge, and code.