Geometric Fragmentation: Exposing Suppressed Knowledge in Large Language Models

Title

Geometric Fragmentation: Exposing Suppressed Knowledge in Large Language Models

Meta Description

Proposing a geometric fragmentation framework to detect suppressed knowledge in LLMs via activation space analysis, distinguishing true knowledge from ignorance and evaluating unlearning robustness.

Article Content

Do large language models (LLMs) truly understand a concept? Traditional behavioral evaluation methods can be deceived by the "suppression" phenomenon: a model's refusal to answer does not mean it lacks recoverable knowledge. A study titled The Geometry of Ignorance: Detecting Suppressed Knowledge in LLMs through Activation Fragmentation proposes a new framework that detects suppressed knowledge by analyzing the geometric structure of activation space, providing an attack-free proxy metric for evaluating model knowledge.

Limitations of Behavioral Evaluation and the Introduction of a Geometric Perspective

The study points out that relying solely on output behavior (such as answer accuracy or refusal rate) to evaluate model knowledge has fundamental flaws. A model may choose not to answer due to safety alignment or fine-tuning, yet internally it still retains extractable knowledge. To bypass this limitation, the authors introduce the concept of "anon-vectors": for a concept-specific query and its anonymized paraphrase, they compute the difference vector between the model's hidden states. By measuring the geometric consistency of these vectors, one can infer whether the model truly grasps the concept.

Hidden States and Activation Space: In the Transformer architecture, each token's input passes through multiple layers of self-attention and feed-forward networks, with each layer producing a high-dimensional vector representation called a "hidden state." Treating the hidden states of all layers and all tokens as a whole constitutes the "activation space." The activation space is the continuous representation space of the model's internal computations, and its geometric structure (such as vector directions, distances, and clustering patterns) encodes the model's understanding of the input and how it invokes knowledge. Unlike the discrete tokens at the output layer, the activation space retains all intermediate computational information before the model generates the final answer, making it a key window for probing suppressed knowledge.

Entropy and Low-Entropy Clustering: In information theory, entropy measures the disorder of a distribution. In the activation space, if the model has a stable and consistent representation for a concept, the related hidden states will cluster in a local region of the space, forming a low-entropy, high-density cluster; conversely, if the model lacks knowledge of that concept, its activation vectors will be scattered across the space, exhibiting a high-entropy fragmented distribution. Therefore, "low-entropy clustering" and "geometric fragmentation" constitute quantifiable geometric metrics for distinguishing known from unknown concepts.

Knowledge vs. Ignorance: Low-Entropy Clustering and Geometric Fragmentation

The core finding is: true knowledge manifests as tight, low-entropy clusters, while ignorance leads to geometric fragmentation. Across ten instruction-tuned LLMs from three model families, all fragmentation metrics reliably distinguish known from unknown concepts. This means that even if a model behaviorally exhibits ignorance (e.g., refusing to answer), the geometric patterns of its internal activations may still reveal latent knowledge. This geometric analysis method offers a more fundamental perspective on knowledge evaluation.

Instruction-Tuned Models: Instruction tuning is the process of supervised fine-tuning a pretrained language model on large amounts of "instruction-answer" paired data, enabling the model to follow human instructions and respond in a conversational format. This process significantly alters the model's behavioral patterns, including introducing refusal behavior due to safety alignment. Therefore, studying "suppressed knowledge" in instruction-tuned models is particularly relevant: the model may be trained during fine-tuning to remain silent on certain concepts, but knowledge acquired during pretraining may still be preserved in the activation space in the form of geometric structure. The study selects ten instruction-tuned LLMs from three model families precisely to verify the generalizability of the geometric fragmentation metric across different architectures and fine-tuning strategies.

Robustness After Unlearning: The Link Between Geometric Fragmentation and Adversarial Recovery

The study further explores the situation after unlearning. Across four attack types and five unlearning algorithms, there is a correlation between fragmented geometry and adversarial recovery. However, the direction of this relationship depends on the attack method: under some attacks, higher fragmentation makes recovery easier, while under others the opposite holds. This indicates that geometric fragmentation is not simply a "knowledge residue" metric but is deeply coupled with attack mechanisms.

Machine Unlearning: Machine unlearning refers to techniques for removing specific knowledge or data influence from a trained model, commonly used for privacy protection (e.g., deleting user data) or safety governance (e.g., removing harmful knowledge). Typical unlearning algorithms use gradient ascent, preference optimization, or parameter editing to make the model behaviorally "forget" a target concept. However, the robustness of unlearning is a critical issue: attackers may use adversarial prompts (adversarial recovery) to re-induce the model to output forgotten knowledge. If unlearning only suppresses output behavior without eliminating internal representations, the knowledge may be recoverable. It is against this backdrop that the study uses activation geometric fragmentation as an attack-free proxy metric for unlearning robustness.

Layer-Wise Analysis: Differences in Depth Across Attack Mechanisms

Through layer-wise dissection, the authors find that different adversarial mechanisms exploit this geometric structure at different depths, with high attack-specific correlations that are completely invisible in behavioral evaluation. This reveals the complex hierarchical nature of internal knowledge representation in models and demonstrates that a single behavioral metric loses a great deal of information.

A Single-Stage, Attack-Free Robustness Proxy Metric

Based on these findings, the researchers propose a single-stage, attack-free proxy metric for evaluating the robustness of unlearning. This metric captures the parametric knowledge level that output-level metrics cannot reach. A reference implementation is publicly available on GitHub (link). This work provides a new geometric tool for understanding knowledge suppression and unlearning mechanisms in LLMs and may influence future standards for model safety evaluation.

In summary, this study opens a new window for detecting suppressed knowledge through geometric analysis of the activation space, highlights the inadequacy of behavioral evaluation, and offers a practical solution for robustness assessment.

One-line Summary

Proposes activation-space geometric fragmentation metrics to detect suppressed knowledge in LLMs and evaluate unlearning robustness without attacks.

Paragraph Summary

Addressing the issue that behavioral evaluation can be deceived by model "suppression," this study proposes detecting suppressed knowledge by analyzing the geometric structure of activation space. The authors introduce "anon-vectors," computing hidden-state difference vectors between concept queries and anonymized paraphrases, and find that true knowledge manifests as tight low-entropy clusters while ignorance leads to geometric fragmentation. Across ten instruction-tuned LLMs, fragmentation metrics reliably distinguish known from unknown concepts. The study further reveals that after unlearning, the direction of correlation between fragmented geometry and adversarial recovery depends on the attack method, and layer-wise analysis shows different attack mechanisms exploit geometric structure at different depths. Based on this, the authors propose a single-stage, attack-free proxy metric for unlearning robustness, offering a new tool for model safety evaluation.