Representational Simplicity ≠ Smaller Circuits: Adversarial Training Reveals a Faithfulness Threshold in Interpretability

Mechanistic interpretability research has long operated under an implicit assumption: if a model's internal representations are "cleaner" — more easily decomposed by sparse autoencoders (SAEs), with more concentrated feature attributions — then its computational processes should be easier to reverse-engineer, and the corresponding causal circuits should be smaller and more tractable. A newly released arXiv paper (arXiv:2609.35890v1) puts this assumption to a direct test, with a surprising conclusion: representational simplicity and causal circuit simplicity do not always move in tandem — they diverge at a specific "faithfulness threshold."

A Widely Assumed but Untested Hypothesis

In recent years, SAE decomposability and concentrated feature attributions have increasingly been treated as evidence that "a model's computation is easier to reverse-engineer." Researchers tend to assume that sparser, more decomposable representations imply simpler underlying mechanisms. But the paper's authors point out that whether this "cleanness in representation and attribution" actually translates to smaller or more tractable causal circuits remains an open question.

Representational tidiness and circuit simplicity are fundamentally properties at two different levels. The former concerns how a model encodes information; the latter concerns the scale of the causal structure needed to recover model behavior. Using the former as a proxy for the latter has long lacked rigorous empirical support. This paper claims to be the first controlled empirical test of whether representational/attribution simplicity translates into causal circuit simplicity at the circuit level.

Sparse autoencoders (SAEs) are unsupervised learning tools trained to decompose a neural network's activation vectors into a set of sparse, interpretable feature directions. The core idea draws from the "superposition hypothesis": neural networks may linearly superimpose thousands of concepts within a smaller activation space, and SAEs use sparsity penalties to force reconstruction using as few activated features as possible. If a model's representations can be reconstructed by an SAE with high quality and low feature activation counts (i.e., sparsely), researchers typically treat this as evidence that the model "uses more modular, separable internal concepts." In recent years, institutions like Anthropic have placed SAEs at the center of mechanistic interpretability research — but most of this work focuses on "discovering interpretable features" rather than verifying whether feature sparsity corresponds to causal circuit complexity.

Adversarial Training as a Controlled Experimental Tool

The paper's methodological design is notably clever. Rather than comparing two arbitrary models, the authors use adversarial training as a controlled experimental lever — because adversarial training can reliably reshape a model's internal representations, without that reshaping itself constituting a test of circuit size.

Starting from the same pretrained GPT-2 Small checkpoint, the authors apply two conditions: matched standard continued training and adversarial continued training. To ensure fair comparison, both conditions require models to maintain performance on the Indirect Object Identification (IOI) task and pass independent robustness validation before entering mechanistic comparison. This "capability-matched + robustness-validated" prerequisite prevents differences in model capability from confounding the conclusions.

The authors operationalize "reverse-engineering complexity" as: the causal structure required to recover model behavior at a fixed faithfulness level. This provides a quantifiable, comparable metric for circuit size.

The Indirect Object Identification (IOI) task is a classic benchmark in mechanistic interpretability, first systematically studied on GPT-2 by Wang et al. (2022). The task format is: given the sentence "Mary and John went to the store. John gave a bottle to ___", the model must predict "Mary" rather than "John." The task's value lies in having a relatively well-understood "ground-truth circuit" — composed of specific attention heads and information flow pathways — about which researchers have reached reasonable consensus. This makes it an ideal controlled setting for comparing reverse-engineering difficulty across models: the task is concrete, behavior is quantifiable, and it requires the model to coordinate information across multiple layers, sufficient to expose structural differences in circuits under different training regimes. Adversarial training here serves to systematically perturb internal representations without changing task performance, creating a pair of models that are behaviorally equivalent but representationally distinct — providing clean experimental conditions for comparing causal circuits.

Three Dimensions of Cross-Comparison

The paper compares the two models along three complementary axes:

• SAE decomposability: Whether the model's representations can be cleanly decomposed by sparse autoencoders;
• SAE feature participation in task attribution: How many SAE features are recruited when completing the task;
• Faithful circuit size: The number of circuit edges recovered from the original computation graph at a given faithfulness level.

At the representational level, the results are intuitive: the robust (adversarially trained) model is more easily decomposed by SAEs, and recruits fewer SAE features in task attribution. In other words, adversarial training does make representations "cleaner." This trend holds not only in the main comparison pair but also generalizes across a seven-point sweep and a second corpus, suggesting the representational simplicity finding is quite robust.

Key Finding: Circuit Size Divergence Is Threshold-Dependent

The truly counterintuitive result comes from the circuit size analysis. The paper finds that circuit size is regime-dependent:

• On the capability-matched IOI task, when faithfulness is below 85%, the standard-trained model's circuit is smaller than or comparable to the robust model's;
• But in the high-faithfulness regime (90%, 95%), the robust model requires significantly fewer circuit edges.

In other words, if you only require rough recovery of model behavior (lower faithfulness), the representationally "cleaner" robust model holds no advantage — and may even be more complex. Only when you demand high-fidelity recovery of model behavior does representational simplicity translate into a smaller causal circuit. The relationship between representational simplicity and circuit simplicity is not a simple linear correlation — there is a clear threshold boundary.

Circuit faithfulness measures: when all components of the model outside the target circuit are replaced with a baseline (e.g., zeroed out or mean-ablated), how well can the remaining circuit reproduce the original model's full output distribution? A faithfulness of 85% means the circuit alone can account for roughly 85% of the original behavioral variance; 95% requires nearly complete recovery of the model's computation on that task. There is no universal standard for choosing a faithfulness threshold — different papers use values ranging from 70% to 95% — which is precisely why the paper's finding that "conclusions depend on the faithfulness regime" is so significant. It reveals that hidden methodological inconsistencies within the field may be masking completely opposite conclusions. High faithfulness requirements are essentially closer to complete reverse-engineering, while low faithfulness resembles finding a coarse-grained functional approximation; the two can impose very different demands on circuit size.

Implications for Interpretability Research

This result has direct methodological implications for current mechanistic interpretability research. Using SAE decomposability as a proxy for "smaller, more easily reverse-engineered circuits" may be misleading in low-faithfulness settings — representationally clean models do not necessarily correspond to simpler causal mechanisms.

The findings also suggest that evaluating interpretability requires explicitly specifying "at what faithfulness level the comparison is being made." Different faithfulness thresholds can yield completely opposite conclusions. For practitioners hoping to obtain "more interpretable" models through adversarial training, this means the benefits may manifest primarily in scenarios requiring high-fidelity explanations, rather than as a general improvement in interpretability.

It should be noted that these conclusions are built on the specific setting of GPT-2 Small and the IOI task, with the main circuit size patterns derived from a single primary comparison pair. Whether they generalize to larger models and more complex tasks remains to be verified. But as the first controlled empirical test of this question, the paper provides a valuable counterexample to the long-standing default assumption that "representational simplicity equals mechanistic simplicity."