Was the Neural Network Always "Clean"? Interpreting Goodfire's Long-Form Explainability Essay

Explainability research has long been overshadowed by one assumption: the interior of a neural network is "a tangled mess," with neurons entangling unrelated concepts that must first be disentangled before anything can be understood. In a new essay, Goodfire researcher Nick Camerata proposes a reverse working hypothesis—neural networks may have been clean all along. If true, we might be able to read the network directly, rather than first performing complex decomposition.

Starting from the "cat-car neuron": was the evidence for polysemanticity misread?

Early explainability research started with individual neurons and found plenty of clean edge detectors and curve detectors. But one famous counterexample is a neuron that appears to activate strongly for both cat faces and the front of cars—a phenomenon known as polysemanticity. The mainstream explanation is superposition—when neurons are insufficient, the network folds together unrelated concepts that rarely co-occur, so a neuron's content is an arbitrary "knot."

Camerata re-examined neuron 4E55 in Inception V1. He spent a few seconds asking "what one thing could it represent?" and the answer surfaced almost immediately: it's not a cat-car neuron, it's a V-shaped neuron. The strongest activations correspond to V-shaped flames, V-shaped fox faces, and V-shaped jets of water spraying from fireballs; the negative pole corresponds to the circular O. Cats and cars were never the point—shape was.

Superposition is a hypothesis systematized by Anthropic and others around 2022: when the number of features a network needs exceeds the available neuron dimensions, the network compresses multiple sparsely co-occurring features onto the same set of neurons, coexisting via approximately orthogonal encodings. It's similar to compressed sensing with fewer microphones than signals—as long as the signals are sparse enough, recovery is still possible. The superposition hypothesis predicts that a neuron's activation is a linear mixture of multiple features, so a single neuron often "can't say clearly" what it represents. It is also the theoretical starting point for decomposition methods like sparse autoencoders (SAEs): if superposition truly exists, the mixture must first be untangled. Camerata's argument is essentially questioning this premise—polysemanticity may be an artifact of the observation tool or sampling bias, rather than the network's true organizing principle.

Recursive generalization: generalization is fractal

Another cited piece of "circuit" evidence is a car feature hidden in dog neurons. But Camerata points out that nothing downstream reads that combination; cars and dogs are placed together because they share pose—Inception V1 has hundreds of such pose neurons. Feature visualization tools themselves are also misleading: they follow the path of least descent, often landing on cats, foxes, or dogs, creating microscope error.

While training a network, he watched a neuron form: at first it responded only to yellow flowers, then expanded to every facet of the photo—other landscapes, green objects like balls, and even dogs with no flowers. Something similar happens in language models: a neuron sensitive to the letters IY (as in Miyazaki) expands to DIY, then to the phrase do-it-yourself, which contains no IY at all. Camerata calls this phenomenon recursive generalization: generalization is not local, but unfolds fractally, layer by layer.

The two stories make opposite predictions: generalization groups similar things together and stays local; superposition packs unrelated things together, which could land anywhere. So this can be tested experimentally. Compress the 480 neurons of Inception 5-1 down to 10: if superposition were true, you'd expect overlapping mixtures of curves + spirals + spiderwebs; in reality the neurons stay clean, just coarser—a single "curves and loops" neuron replaces the original 10 curve detectors. Less space, coarser concepts; more space, finer concepts.

Inception V1 is the classic convolutional network proposed by Google in 2014. Because of its breakthrough on ImageNet and its relatively compact structure, it has long been the "fruit fly" model of explainability research. Its convolutional layers (such as conv5-1, i.e., the 480 channels of Inception 5-1 mentioned in the text) were systematically studied by Olah et al. using feature visualization, and much of the early evidence about neuronal polysemanticity comes from here. Camerata chose it as the object of re-analysis because there are publicly available neuron visualization libraries to compare against, and because it allows direct comparison with conclusions from eight years ago. Understanding this is important: "where vision models were eight years ago" in the text refers precisely to the Inception-era research paradigm, and language models may be at the same stage now—just larger in scale and with weaker tools.

Language models: from 0/14 to 90% single-duty

Language models are larger and stranger—does this conclusion still hold? In 2023, Anthropic's study of small models gave a median neuron explainability score of 0/14; OpenAI's automated interpretability project found that about 1.7% of neurons were explainable. But redoing this with better tools and modern models paints a completely different picture. A preliminary audit of Qwen3-32B shows that about 90% of neuron poles have only one primary duty, about 7% have two related duties, and only 2-4% remain unexplained; similar results can be reproduced on trillion-parameter-scale models.

Language neurons also behave unexpectedly. A seemingly polysemantic "comma neuron" has its highest activations on a bunch of unrelated commas; on closer inspection, it is tracking Colorado—the comma appears in contexts where the model is generating that word. This neuron maintains a rolling summary of the generated context; turn it up, and the model plans a hike. Another neuron activates on the letters PR in base64 URLs and on Bengali text; after decoding the URL and transliterating the Bengali, both are PR—not a mixture, but the same thing in disguise.

Explainability has crossword-puzzle-like momentum: at first, a neuron can only be explained by other neurons you can't yet read, but as the map fills in, each answer constrains the next. When you can only see 2% of the network, you can't tell where the tool is broken; when you see most of it, the unexplained residue points directly to tool problems.

Sparse autoencoders (SAEs) are a class of unsupervised decomposition tools: they train a bottleneck network with a sparsity penalty to reconstruct model activations as a linear combination of many sparse features, attempting to split superposed neurons into "monosemantic" features. Anthropic's 2023 use of SAEs to extract interpretable features from language models was seen as a milestone in explainability. But Camerata points out the risk: if the original network is already clean, the SAE will forcibly split a single concept into multiple components, manufacturing polysemanticity that doesn't exist—like splitting a V-shaped neuron into cars and cats. The effectiveness of SAEs depends on the superposition hypothesis, so it is both a tool and an object of testing for that hypothesis; it cannot be used to circularly argue that superposition exists.

From neurons to circuits: diagnosis, retrieving money, and the definition of "clean"

This clarity can extend to the circuit level. In Qwen, a medical diagnosis passes through only a four-neuron hub; remove them, and the model instead asks about itching. In the trillion-parameter Kimi K3, a "found a wallet" prompt lights up neurons related to lost-and-found and honesty, and the output is "return it."

Being "clean" does not mean every neuron matches labels we already have, but that every component is designed to be read and built upon by later components. It's like how the financial district is barely an issue in a small town, but can be found on a New York map in a minute. Disassembling a network that was never a knot may instead make it look like one—like reconstructing Brooklyn from 70 fragments of Tokyo. This is exactly what sparse autoencoders do to the V neuron: splitting it into a car feature and a cat feature manufactures polysemanticity that never existed.

Camerata contrasts two styles of explainability: one builds a large feature list and then audits it; the other starts from a question and zooms in and out like a map. Their errors are asymmetric: if you assume the network is clean and you're wrong, you get a wrong map, and messy neurons are clues; if you assume the network is packed and you're wrong, messy neurons are the endpoint, because you stop looking.

This may also be the "bitter lesson" of explainability: scaling up ordinary investigation beats cleverly designed decomposition. The author cautiously states that this is only a working hypothesis held with about 85% confidence, and not everything has been audited. But frontier models may be where vision models were eight years ago: readable, just needing many small corrections. If true, explainability will be far more tractable than we thought—not a knot, but a city, or a rainforest, where the network has already done almost all the self-preparation work.

The full essay and interactive microscope are available at alreadyclean.goodfire.com.

Automated interpretability refers to using another language model to generate natural-language explanations for neurons or features, then scoring them by how accurately they predict simulated activations; OpenAI's 2023 work falls into this category. The "about 1.7% of neurons explainable" and "median 0/14" in the text are both products of this kind of scoring. The limitation of such methods is that the explanations are model-generated, the scoring depends on correlation between explanation and activation rather than causal intervention, and early models had small capacity and crude tools, making it easy to underestimate explainability. Camerata emphasizes that results reverse when redone with "better tools and modern models," suggesting that early low scores may have been mainly measurement error rather than the network being inherently messy.

One-line Summary

A Goodfire researcher proposes a working hypothesis: the interior of neural networks may have been clean all along, and polysemanticity is a misreading.

Paragraph Summary

Goodfire researcher Nick Camerata proposes a reverse working hypothesis: the interior of neural networks may have been "clean" all along, rather than a tangled knot. He re-examines the famous "cat-car neuron" 4E55 and finds it is actually a V-shaped neuron—cats and cars were misreadings of shape; he also proposes "recursive generalization"—generalization has a fractal structure, rather than arbitrary packing caused by superposition. A preliminary audit of Qwen3-32B shows that about 90% of neuron poles have only one primary duty, and only 2-4% remain unexplained, in stark contrast to earlier explainability scores of 0/14. He also warns that sparse autoencoders may split clean neurons into polysemanticity that doesn't exist. The author self-rates the hypothesis at about 85% confidence and believes that scaling up ordinary investigation may beat clever decomposition.