Neural Networks Were Always Clean? Goodfire Researcher Overturns Superposition Theory

Over the past decade, "AI is a black box" has become almost an unquestioned consensus. Billions of parameters stacked layer upon layer, and no one can quite explain what's happening inside. But the foundation of that consensus—Superposition—is now being overturned by one of the very people who proposed it.

Goodfire researcher Nick Camerata published a long blog post with a provocatively blunt title: Neural Networks Were Always Clean. His core argument: for the past decade, we've been reading it wrong.

Superposition: A "Bleak" Theory That Ruled the Field for Years

A decade ago, scientists used feature visualization techniques to illustrate what each neuron most wanted to see, and discovered something odd: one neuron responded strongly to both cat faces and car fronts. Even more striking was Inception V1—the "model organism" of interpretability research at the time—whose final layer was almost entirely populated by these chaotic neurons, with only about 0-5% having a clearly explainable purpose.

Thus Superposition was born: neurons weren't enough, so the network crammed unrelated features into the same neuron—since they don't usually conflict in practice. Like a tangled ball of yarn with a knot: messy but usable. This theory had a rather bleak corollary: if the network is a knot, you can't study it directly. You must first train a second model (like a sparse autoencoder) to untie the knot, then study the untangled version. The original? Unreadable. Throw it out.

One of the people who proposed this theory was Nick Camerata himself. He was involved in OpenAI's early interpretability work and led OpenAI's first large-scale project to have AI interpret its own neurons—at the time only about 2% of neurons were understood, which was one reason he once believed in superposition.

Sparse Autoencoder (SAE) is an interpretability tool that rose to prominence after superposition theory became popular. The idea: since the activations of raw neurons are dense vectors with multiple features superimposed, train an autoencoder with a sparsity constraint to decompose high-dimensional activations into a large number of nearly monosemantic sparse features. SAEs became the dominant method in interpretability around 2023, with institutions like Anthropic using them to extract large numbers of readable features from large models. But critics point out that SAEs are essentially post-hoc "decomposers"—the features they reconstruct may not correspond to the actual computational units inside the network, and the sparsity constraint itself introduces artificial bias. Camerata's "read the network directly" argument is a direct challenge to the SAE paradigm: if raw neurons are already sufficiently monosemantic, adding an intermediate decomposition layer may actually scramble the signal.

First Strike: What Do Cat Faces and Car Fronts Actually Have in Common?

Camerata says he stared at that cat-face/car-front neuron image hundreds of times, but this time he asked himself for the first time: what do these two actually have in common? The answer jumped out in five seconds—the V shape. Cat faces are V-shaped, car fronts are V-shaped, V-shaped flames, V-shaped water spray from fireboats, fox faces—all V's.

Now look at this neuron's "abs"—what doesn't it like? An O. It fires wherever there's a V, and the opposite for O's. A V-shape detector that also fires for flames—that makes perfect sense. What about those misleading cat whiskers? They're hitchhikers brought in by the visualization technique: once the optimization walks down the cat path, the whiskers come along for the ride, even though the neuron doesn't actually want them.

The 2019 experiment once considered the only evidence of a superposition circuit—where a car feature appeared to be hidden inside a bunch of unrelated dog neurons—he re-examined and found it wasn't about hiding car and dog features at all; it was about pose being grouped together. Inception V1 has over 100 of these cross-cat/dog/car pose neurons that no one had noticed before. The reason it looked like hiding back then was partly that the visualization method had been reward-hacked—the optimization process fell into a dog basin and couldn't climb out. In his words: this is the only superposition circuit I've ever seen, and it's wrong. As far as he knows, no second circuit like it has ever been published.

Second Strike: From 2% to 99% Readability

Overturning old evidence isn't enough—he prepared a three-pronged attack. The first strike targets Inception V1's final layer—the main evidence for superposition theory back then. He did three things:

• First, fix the microscope: patch the feature visualization, raising the proportion of understandable neurons in the same batch from about 0-5% to about 50%
• Retrain Inception V1 with its original architecture plus modern regularization: about 70%
• Switch to a modern network with the same parameter count and data (with residual connections and LayerNorm): about 99% of neurons in the final layer are understandable

He estimates that the chaos in that layer back then wasn't superposition—a novel learning principle—but rather bad learning: flawed architecture plus weak tools. Ironically, the architectural improvements the field made to help deep layers learn better incidentally fixed interpretability too, with no one aiming for it.

The second strike targets language models. His own automated project in 2023 only understood about 2% of neurons in GPT-2; re-examining now, about 70% appear to have a single meaning at first glance, with most of the rest having two or three related meanings. Even more striking are preliminary audits of modern large models like Qwen 32B: about 90% of neurons have only one "homepage," about 7% have two related "subpages," and only 2-4% remain incomprehensible. He repeatedly emphasizes that this number depends heavily on how you count—change the threshold and it changes.

For trillion-parameter models, he also produced attribution graphs: the model encounters a wallet, and wallet, cash, and owner's ID connect all the way to mistake and honesty, ultimately answering "return it." As far as he knows, this is the first publicly published attribution graph of a trillion-parameter model. On Mixture-of-Experts (MoE) models, his experts were highly specialized—one expert almost exclusively handled RPG tabletop game bosses. This is practically an anti-superposition architecture: trying to cram ten thousand unrelated things into the same expert is a non-starter, since the router is just a small linear layer and simply can't route them.

The third strike is stress testing. According to superposition theory, when neurons are scarce, features should stack. He cut a layer of Inception V1 from 480 neurons down to 10 and retrained, to see if it would stack. It didn't stack—it just got coarser: 10 curve detectors became one curve-and-loop neuron, several meta-neurons became two meta-neurons. No stacking of curves plus orange spirals plus spider webs—just coarser generalization. Even compressing the residual stream to 46 dimensions, what he saw was only coarsening, not carefully woven interference.

After the three strikes, his judgment: superposition should be demoted from a default assumption to a last-resort hypothesis. When something is incomprehensible, it's more likely that the tools are inadequate, the architecture is sick, or the following reason applies.

Attribution graph is key to understanding the "trillion-parameter model" experiment in the article. It represents the model's computation as a directed graph: nodes are intermediate activations or features, edges indicate causal influence strength, typically obtained through gradient attribution or activation patching on specific inputs. Unlike looking at individual neurons, attribution graphs trace how information flows step by step from input features (like "wallet," "cash," "owner's ID") to the final output (like "return"), thus exhibiting a readable reasoning chain without decomposing the network. What Camerata means by "zooming like Google Maps" is locally expanding or globally overviewing such graphs as the question demands, rather than pre-enumerating all features and auditing them one by one.

Recursive Generalization: Neurons Are Living Trees, Not Dead Knots

What exactly is that remaining polysemy? He coined a new term: recursive generalization. Imagine a tree: as a neuron grows, it branches along related directions, sprouting twigs, which sprout more twigs. Locally, every step makes sense; globally, it's no longer monosemantic.

One neuron initially only recognized "iy" (as in Miyazaki's "iy"), during training grew to include "diy," and finally extended to the phrase "do it yourself," where even the "iy" is gone. Another geological neuron branched out to Cesium—a JavaScript library for geology.

Why is this so much better than superposition? Superposition is cramming—features are shoved together because they don't conflict, and the reason for cramming teaches you nothing. Recursive generalization is growth: every branch has a lineage, can be traced, can be predicted, and can be watched growing in real time in training recordings. One is a dead knot, the other a living tree—completely different worlds.

He also explains why networks naturally love to generalize, calling it the "circuit race." Train on a photo of a Labrador from scratch: one circuit recognizes broad features like green grass and brown fur, another circuit counts the color of the 322nd pixel—a dead feature. The broad one gets more gradient at every step, grows faster, grabs all the error signal, and the narrow one starves to death. Generalization isn't a second step—it starts from the very first step. The winners are always the broad, reusable ones.

So "neural networks are clean" doesn't mean every neuron corresponds to a human concept—that would actually be bad news, indicating the network hasn't learned anything beyond our imagination. Clean means the network is genuinely learning, forming an ecosystem, like a rainforest: every animal has a lineage, every niche has a reason. Why are tigers orange? Because deer can't distinguish orange from green, and growing green fur is too hard for mammals. Once you know this, an entire class of animals makes sense.

Circuit race borrows the metaphor of "competitive growth" from biological development and evolution. In early training, multiple candidate circuits simultaneously try to fit the same batch of error signals. Whichever has broader, more reusable features gets gradient on more samples, thus growing faster and grabbing more parameter capacity, ultimately starving narrow circuits. This mechanism explains why networks generalize from the very first step rather than overfitting first and generalizing later: generalization isn't a post-hoc effect of regularization but an initial bias of gradient competition. It also explains why modern architectures (residual connections, LayerNorm) often incidentally improve readability—these designs make it easier for broad features to propagate gradients and harder for narrow features to survive, so networks naturally grow into cleaner ecosystems.

Methodological Shift: Read the Network Directly, Like Using Google Maps

Methodologically, he advocates not decomposing first but reading the network directly. He calls this exploratory interpretation—like Google Maps, freely zooming on attribution graphs, letting the question lead you, rather than starting with an exhaustive feature list and auditing it. He says the network itself is the answer; interpretability is about finding the right questions. Chop Tokyo into pieces and scatter it across Brooklyn, and Tokyo will only look messier; that's exactly what sparse autoencoders do—splitting micro-neurons into car features and cat features, moving further from the truth.

The implications for AI safety are exciting. If the network is a well-planned city rather than a tangled knot of yarn, then deception can't hide—deception must be accessible to the network's own readers (downstream neurons), otherwise it can't propagate. We just need to become another reader to read it. He says what he truly fears isn't missing some activation, but that it's right in front of us and we don't know how to read it. All the misreadings of the past decade were failures to understand what was right under our noses—not once did a hidden activation slip past.

He elaborates on the city metaphor: in a small town, asking a few questions won't tell you where the financial district is—banks on one street, insurance companies on another, accountants working from home. But in New York, you can locate it on a map in a minute. Across three models differing by 300x in size, they found that finding the hammerhead-dog neuron is actually faster in larger models, because beneath it sits a whole family of pale wavy side-of-head ear neurons dragging it along—bigger is easier to navigate.

A Few Honest Caveats

This is Goodfire's company blog, not a peer-reviewed paper; the accompanying technical paper hasn't been published yet. All numbers are estimates and depend heavily on how you count. The author himself says he's only 85% confident—note, 85% confident that the entire worldview is correct, not 85% of neurons are correct—these are two very different kinds of confidence. He admits the toy-model experiments back then weren't wrong per se; the disagreement is only about which one the real world actually is. And, Goodfire is a company that makes interpretability tools—the optimistic narrative aligns with their business direction.

But if he's right, the prospects for interpretability are far better than the entire field believes. Let me close with his words: we've always had the best laboratory in the natural sciences—every neuron readable, every intervention doable, every experiment rerunnable on an identical copy. What was missing was simply enough automated scientists to watch it for us.

One-line Summary

One of superposition theory's originators overturns the old view, arguing neural networks were always clean and readable.

Paragraph Summary

Goodfire researcher Nick Camerata—one of the originators of superposition theory—published a long post overturning his own prior theory, arguing that neural networks are not chaotic superpositions but "clean" structures with discernible patterns. He reinterprets classic evidence: the supposed cat-face/car-front neuron's commonality is the V shape, the experiment treated as superposition circuit evidence is actually about pose neurons, and the visualization method itself was misleading. Through a three-pronged approach—fixing visualization tools, retraining with modern architectures, and stress testing—Inception V1's final-layer readability rose from about 0-5% to about 99%, and audits on GPT-2 and Qwen 32B also show the vast majority of neurons have only one "homepage." He proposes "recursive generalization" to replace superposition: neurons grow and branch like living trees, with every branch traceable. Methodologically, he advocates reading the network directly and interpreting exploratorily like using Google Maps, rather than decomposing and auditing first. If correct, deception can't hide—hugely significant for AI safety.