Circe: A Training-Free LLM Self-Evaluation Method Using Component Interaction Consistency

Large Language Models (LLMs) are being deployed in an growing number of real-world applications, and determining whether their responses are correct has become key to reliable use. However, in practical scenarios there is often no reference answer available for comparison, which makes evaluation tricky. Existing reference-free evaluation methods mainly rely on output signals, but output signals may give high scores to incorrect answers and do not necessarily reflect correctness. To address this, researchers propose a training-free self-evaluation method called Circe, which fuses internal computation signals with external output signals to evaluate response correctness without relying on reference answers or repeated sampling.

Limitations of Existing Methods: Looking Only at Outputs or Only at Hidden-State Trajectories

A common approach to reference-free evaluation is to analyze the probability distribution of model outputs, such as perplexity or negative log-likelihood. These signals are simple to compute, but they have a fundamental problem: the model may be equally "confident" about incorrect answers, causing output signals to become decoupled from correctness. In recent years, some studies have attempted to mine the model's internal states and obtain supplementary signals by analyzing the trajectories of hidden states across layers. However, these works mainly focus on layer-by-layer changes in hidden states, while hidden states record the joint updates of attention and feed-forward networks (FFN), making it impossible to distinguish what each of these two components contributes and ignoring how they interact.

This is precisely the starting point of Circe: rather than looking only at the combined trajectory of hidden states, it decomposes the interaction process between attention and FFN to extract finer-grained correctness signals.

Perplexity and negative log-likelihood are two commonly used metrics in language model evaluation and are closely related. Negative log-likelihood (NLL) refers to the negative average log probability that the model assigns to each token in a given text; the lower the value, the more the model "believes" the text is reasonable. Perplexity is the exponential form of NLL (perplexity = exp(NLL)) and can be understood as the model's average degree of uncertainty when predicting the next token. Both rely only on the probability distribution of model outputs and are easy to compute, but their common flaw is that they measure the model's "familiarity" with or "fluency" of the text, not factual correctness. A model may well assign low perplexity to a fluent but incorrect answer, which is the root cause of output signals becoming decoupled from correctness.

In the Transformer architecture, each layer consists of an attention sublayer and a feed-forward network (FFN) sublayer connected in series, both of which are added to the hidden state through residual connections. The attention mechanism is responsible for aggregating information across tokens, while the FFN performs a nonlinear transformation independently at each position. The layer-by-layer changes in hidden states are actually the superimposed result of updates from these two components, so observing only hidden-state trajectories cannot distinguish whether a change in a given layer comes from attention or from the FFN. The core insight of Circe is precisely this: by separately tracking the direction of attention updates and the direction of FFN responses, and examining whether the two are aligned, it can extract fine-grained signals that are masked by the mixed trajectory.

Core Innovation: Component Interaction Consistency (CIC)

The first internal signal proposed by Circe is Component Interaction Consistency (CIC). It measures the degree of alignment between the direction of attention updates in early layers and the direction of attention-induced FFN output changes in later layers. Intuitively, if the model's internal computation is coherent when generating a response, then the information passed by the attention module and the adjustments made by the FFN accordingly should be aligned in direction; conversely, if the two directions diverge, it may suggest contradictions or instability in the reasoning process.

The value of CIC lies in the fact that it moves beyond the coarse-grained perspective of the "overall hidden-state trajectory" and directly characterizes the quality of collaboration between the two core components, attention and FFN. This component-level interaction signal is something that previous methods based on hidden-state trajectories failed to capture.

Supplementary Signals: Representation Geometry and Likelihood Confidence

In addition to CIC, Circe also introduces Representation Geometry (RG) to characterize the geometric features of hidden-state trajectories, specifically including normalized displacement and angular change. This signal describes, from another angle, whether the evolution of the model's internal representations is smooth and reasonable.

On the external signal side, Circe converts the average negative log-likelihood of generated tokens into a bounded Likelihood Confidence (LC) score. Compared with directly using negative log-likelihood, the bounding treatment makes this signal more stable when fused with other scores.

Ultimately, Circe fuses the three signals CIC, RG, and LC into a unified correctness score. The entire method requires no training, no reference answers, and no multiple sampling, giving it strong practicality.

Representation Geometry focuses on the geometric properties of hidden states as they move across layers. Normalized displacement measures the relative magnitude of change between hidden-state vectors in adjacent layers, while angular change characterizes the degree of deflection in the direction of movement. If the model's internal representation evolves smoothly, the displacement between adjacent layers should maintain a stable direction and moderate magnitude; if there is severe directional deflection or abnormally large displacement, it often means that the model's processing at that position is not sufficiently stable. Such geometric features are unrelated to output probabilities and belong to purely internal signals, serving as another dimension of supplementary evidence beyond CIC.

Experimental Validation: Generalization Across Models, Domains, and Modalities

The researchers conducted experiments on five LLMs and six benchmarks covering four domains. The results show that, compared with existing baselines, Circe can more effectively distinguish correct from incorrect responses. Even more noteworthy is its generalization: the method is also effective in multilingual, multimodal, and open-world settings. This means that Circe is not limited to a specific model or task type and has broad applicability potential.

From a methodological perspective, Circe provides a valuable idea: LLM self-evaluation does not have to focus only on the final output, nor does it have to rely only on coarse-grained analysis of hidden states; digging into the interaction consistency between components can uncover more reliable correctness signals. For teams that need to deploy LLMs in scenarios without labeled data, this type of training-free evaluation method is worth attention.

As LLM application scenarios expand, how to evaluate model outputs at low cost and with high reliability will remain an ongoing need. Circe opens a new path for training-free self-evaluation by approaching it through component interaction consistency.

One-line Summary

Circe proposes a training-free self-evaluation method that assesses LLM response correctness by fusing component interaction consistency with output signals.

Paragraph Summary

Circe is a training-free self-evaluation method for large language models, designed to determine whether a response is correct without reference answers or repeated sampling. Existing reference-free methods either rely on output probability signals (such as perplexity), which may give high scores to incorrect answers, or analyze layer-by-layer hidden-state trajectories, but cannot distinguish the respective contributions of attention and FFN. The core innovation of Circe is Component Interaction Consistency (CIC), which measures the degree of alignment between the direction of attention updates in early layers and the direction of attention-induced FFN output changes in later layers. It also fuses Representation Geometry (RG) and bounded Likelihood Confidence (LC) into a unified score. Experiments on five LLMs and six benchmarks show that the method outperforms existing baselines and generalizes to multilingual, multimodal, and open-world scenarios.