Neural Networks Are Already "Clean": A Paradigm Shift in Interpretability Research

A Counterintuitive Proposition: Networks Don't Need to Be "Fixed"

Interpretability research has long assumed a premise: the internals of neural networks are messy, redundant, and entangled, and the researcher's job is to act like a surgeon—prying apart tangled neurons, pruning them, and replacing them with human-readable rules. But an article that sparked discussion on Reddit proposes the opposite conclusion: neural networks are already "clean"—what's unclean is the way we observe them.

If this proposition holds, it means that a large body of past interpretability work aimed at "eliminating polysemy" and "disentangling features" may have been solving a pseudo-problem all along. The real bottleneck isn't inside the model, but in the language and tools we use to describe it.

The Old Paradigm: Treating the Network as an Object That Needs Cleaning

The traditional toolbox of interpretability research is full of "surgical" operations:

• Neuron-level analysis: Searching for "grandmother neurons" that correspond to a single semantic concept, and pivoting to circuit analysis after failing to find them
• Sparse autoencoders (SAEs): Decomposing superimposed activations into sparse features
• Pruning and ablation: Removing a component and observing whether performance degrades
• Concept activation vectors (CAVs): Attempting to find linear directions for abstract concepts

These methods share an implicit assumption: there is "contamination" in the network's representation space that requires external intervention to restore order. Superposition is often cited as the core evidence—models encode far more features than their dimensionality by using high-dimensional space, causing every neuron to participate in multiple concepts.

The author's challenge is: Does superposition really mean messiness? Or is it simply a geometric structure for which we haven't yet found the right coordinate system?

Superposition is the key background for understanding this debate. It refers to the following: when a neural network needs to represent far more features than its neuron dimensionality, the model tends to encode multiple features using "approximately orthogonal" high-dimensional directions rather than allocating an independent neuron to each feature. This is similar to squeezing more "directions" out of a two-dimensional plane using multiple non-perpendicular vectors—at the cost that each direction carries a slight projection of the others, making a single neuron appear to "handle everything."

Superposition has long been viewed as "contamination" because it violates the human-preferred intuition of "one unit, one concept." But from a linear algebra perspective, as long as the interference between feature directions is small enough, the model can still read out any single feature with high precision via dot products. Thus superposition itself does not equal information chaos; it is more like an efficient compression strategy under capacity constraints—the only question is whether we have found the basis transformation that can "align" this set of directions with the coordinate axes.

A New Perspective: Cleanness Is an Observer-Relative Property

The article's core argument can be summarized with an analogy: gas molecules appear to collide randomly at the microscopic level, yet macroscopic quantities like pressure and temperature are extremely stable. No one thinks gas "needs to be cleaned," because we have the right statistical language to describe it.

Neural networks may be in a similar situation. The polysemy of a single neuron may, under a more appropriate coordinate system, manifest as clear, separable structure. The interactive visualization accompanying the article attempts to demonstrate this: when switching between different projection methods, originally "entangled" activation clusters exhibit clear stratification and boundaries.

This brings about a methodological shift—from transforming the model to transforming the way we observe. No longer asking "how do we make the network interpretable," but rather "which representation transformation allows the network's interpretable structure to emerge naturally."

Impact on the Existing Toolchain

If "cleanness" is observer-relative, then the positioning of several mainstream techniques needs to be re-evaluated:

Sparse autoencoders may not be "restoring" a contaminated representation, but rather selecting one among many viable coordinate systems that matches human sparsity preferences. Sparsity is a human aesthetic, not necessarily an intrinsic property of the network.

Circuit analysis (such as tracing the collaborative paths of attention heads and MLPs for specific behaviors) remains valid, but it explains computational pathways, not the "cleanliness" of the representation space. These are different levels of questions and should not be conflated.

Probe classifiers' high accuracy is often interpreted as "the concept really exists," but it can also be read in reverse: "there exists a set of coordinates that makes the concept linearly separable"—the latter reading is fully compatible with the new paradigm.

The key conclusion of this section is: old tools need not be discarded, but their philosophical positioning must be downgraded from "repairing the network" to "choosing a readable perspective."

Sparse autoencoders (SAEs) are among the most closely watched tools in interpretability in recent years. Their approach: train an "overcomplete" linear encoder on top of a model layer's activations, mapping the original activations into a higher-dimensional sparse latent space, with an L1 penalty encouraging each activation to be explained by only a few latent variables. Proponents argue this can "untangle" superposition, yielding feature directions that correspond to single semantics.

But from the new paradigm's perspective, SAE's sparsity constraint is essentially a human preference bias—we are accustomed to describing a concept with a few words, so we require the latent space to be sparse too. The true structure inside the network may not be sparse; SAE may simply have found a coordinate system that "happens to match human aesthetics," rather than the one correct decomposition. This does not negate SAE's practical value, but it reminds us not to treat the features it outputs as atomic units the network "originally has."

Controversies and Open Questions

The Reddit discussion was not one-sided. Objections mainly focus on two points:

First, "clean" lacks an operational definition. If any network can be made to appear clean through some transformation, then this property has no discriminating power—it can't be used to compare different models or guide training.

Second, existence does not equal discoverability. Even if a coordinate system that makes structure clear exists, under realistic computational budgets, we may not be able to find it. Mathematical elegance does not guarantee engineering usability.

Supporters respond: many concepts in physics (such as phase space and entropy) initially faced criticism of being "non-operational," but eventually became standard tools. What interpretability may need is precisely such a conceptual upgrade, not more engineering tricks.

Discoverability is an easily overlooked technical difficulty in this debate. Mathematically, "there exists a coordinate system that makes structure clear" and "we can effectively find it" are two different things. The variety of transformations of representation space is nearly infinite, including linear projections, nonlinear embeddings, manifold learning, and more—the search space is enormous.

Even if an ideal coordinate system exists, whether it can be approximated under a finite computational budget via gradient optimization, spectral methods, or sampling strategies remains an open question. More subtly, if this "clean coordinate system" differs for every model and every layer, the cost of searching individually for each scenario could be unacceptably high. Critics therefore point out: a property that holds only at the philosophical level but cannot be stably reproduced in engineering is hard to establish as an organizing principle for interpretability research—unless someone can provide a computable, verifiable criterion.

Implications for Practitioners

For engineers doing model debugging, safety evaluation, or explaining models to non-technical audiences, this paradigm shift offers several mindset adjustments that can be applied immediately:

When you encounter "some neuron that seems to handle everything," first try switching to a different representation before rushing to do attribution or pruning. When evaluating an interpretability method, distinguish whether it is discovering structure or imposing structure. Stay alert to phenomena like "superposition"—they may be real constraints, or they may simply be artifacts of our current observational tools.

The article's interactive visualization is a direct demonstration of this approach: changing the view can sometimes get you to the answer faster than changing the model.

One-line Summary

Neural networks are already clean; the sense of messiness comes from the observational coordinate system, not from contamination inside the model.

Paragraph Summary

The article proposes a counterintuitive proposition: interpretability research has long assumed that networks are internally messy and need to be cleaned like a surgical operation, but the real problem lies in how we observe the network, not in the model itself. The author draws an analogy to gas molecules—random at the microscopic level, yet stable at the macroscopic level, because we have the right statistical language. Phenomena such as superposition and polysemous neurons may simply be geometric structures that appear under the wrong coordinate system; switch to another representation and they may naturally stratify. This shift means: sparse autoencoders, circuit analysis, probes, and other tools need not be abandoned, but their philosophical positioning should be downgraded from "repairing the network" to "choosing a readable perspective." The main objection in the Reddit discussion is that "clean" lacks an operational definition, and existence does not guarantee discoverability. The practical takeaway: when encountering polysemous neurons, first change the representation rather than rushing to prune or attribute.