Circe
A training-free self-evaluation method for LLMs that fuses internal component-interaction consistency, representation geometry, and external likelihood-confidence signals to assess answer correctness without reference answers or repeated sampling.
Timeline (last 90 days)
Circe is a training-free self-evaluation method that assesses the correctness of LLM answers by fusing internal computational signals with external output signals, without relying on reference answers and without repeated sampling.
The Component Interaction Consistency (CIC) signal proposed by Circe measures the degree of alignment between the direction of attention updates in early layers and the direction of attention-induced FFN output changes in later layers.
The Representation Geometry (RG) signal introduced by Circe characterizes the geometric features of the hidden state trajectory, specifically including normalized displacement and angular change.
Circe converts the average negative log-likelihood of generated tokens into a bounded Likelihood Confidence (LC) score, and the bounding treatment makes this signal more stable when fused with other scores.
Circe fuses the three signals CIC, RG and LC into a unified correctness score.
The researchers conducted experiments on five LLMs and six benchmarks covering four domains.
Experimental results show that, compared with existing baselines, Circe can more effectively distinguish correct from incorrect answers.
Circe is likewise effective in multilingual, multimodal and open-world settings.
All Facts (8)
Circe is a training-free self-evaluation method that assesses the correctness of LLM answers by fusing internal computational signals with external output signals, without relying on reference answers and without repeated sampling.
50%UnverifiedThe Component Interaction Consistency (CIC) signal proposed by Circe measures the degree of alignment between the direction of attention updates in early layers and the direction of attention-induced FFN output changes in later layers.
50%UnverifiedThe Representation Geometry (RG) signal introduced by Circe characterizes the geometric features of the hidden state trajectory, specifically including normalized displacement and angular change.
50%UnverifiedCirce converts the average negative log-likelihood of generated tokens into a bounded Likelihood Confidence (LC) score, and the bounding treatment makes this signal more stable when fused with other scores.
50%UnverifiedCirce fuses the three signals CIC, RG and LC into a unified correctness score.
50%UnverifiedThe researchers conducted experiments on five LLMs and six benchmarks covering four domains.
50%UnverifiedExperimental results show that, compared with existing baselines, Circe can more effectively distinguish correct from incorrect answers.
50%UnverifiedCirce is likewise effective in multilingual, multimodal and open-world settings.
50%