Nick Cammarata's In-Depth Essay: AI Isn't a Black Box — 99% of Neurons Are Clearly Readable

A Decade-Long Consensus Overturned by a Single Neuron

In the field of AI interpretability, neuron 4e:55 in the InceptionV1 model is a textbook classic. It is strongly activated by both "cat faces" and "car fronts" — a phenomenon long regarded as ironclad proof that the underlying logic of neural networks is unexplainable.

According to Nick Cammarata, an early mechanistic interpretability researcher at OpenAI, this neuron actually detects a V-shaped geometric form. Whether it's the pointed face of a fox, a jet of flame, the sharp front of a car, or the V-shaped water spray from a fireboat, all trigger its positive pole; all circular contours trigger its negative pole.

This means that the "polysemanticity" evidence cited repeatedly over the past decade is very likely not a true property of neural networks at all, but a misinterpretation caused by crude observation tools.

From "Disassembly" to "Reading": The Battle Over Interpretability Research Directions

The Rise and Cost of the Superposition Hypothesis

When researchers found deep-layer neurons difficult to understand, academia proposed the "superposition hypothesis": when the number of neurons is insufficient, AI folds unrelated concepts into the same neuron, because they rarely activate simultaneously and thus don't interfere with each other.

In 2020, Cammarata and Chris Olah further reinforced this hypothesis with evidence in their milestone paper Zoom In — the network had hidden car-feature signals inside a group of "dog neurons."

Following this logic, labs like Anthropic and OpenAI adopted sparse autoencoders (SAEs) as the mainstream approach, training auxiliary models to forcibly extract multiple features crammed into neurons, reclassify them, and attach human-readable labels.

What is a Sparse Autoencoder (SAE)?

A Sparse Autoencoder (SAE) is an auxiliary neural network, typically consisting of a hidden layer wider than the original activation dimension, trained with sparsity constraints (such as L1 regularization). Its core idea is: assuming that original neuron activations are linear superpositions of several "true features," training an encoder to map activations to a higher-dimensional, sparser space and then decoding back to the original space can "disentangle" features that are tangled together. Each sparse dimension ideally corresponds to a human-readable monosemantic feature, and researchers use this to label neurons as "cat," "car," etc. In a series of works from 2023–2024, Anthropic applied SAEs to Claude's residual stream, and OpenAI also used them to analyze the GPT series, making SAEs one of the most mainstream tools in mechanistic interpretability. The critique in this article points directly at this paradigm: if the "features" extracted by SAEs have broader response widths than native neurons, then they may be manufacturing illusions rather than revealing truth.

What is the Residual Stream?

In the Transformer architecture, the residual stream refers to the "main artery" that runs through every layer and is continuously read from and written to by each sublayer (attention, MLP) — that is, the hidden state vector formed by adding each layer's output to that layer's input. It resembles a shared communication bus: attention heads read information from the residual stream and write results back, MLPs also perform nonlinear transformations on this basis, and information accumulates continuously across layers rather than being overwritten. Precisely because of this, the residual stream is considered to carry the model's most core "conceptual representations," and is also the object most frequently targeted by SAEs. Neel Nanda emphasizes that "SAEs remain irreplaceable on the residual stream" precisely because the residual stream is high-dimensional, information-dense, and repeatedly read and written — the place most likely to exhibit "representation superposition" — and conclusions about native neuron readability may not transfer directly to it. Understanding the read-write mechanics of the residual stream is necessary background for grasping the crux of both sides' debate.

The Reversal Brought by Tool Upgrades

During his renewed research, Nick Cammarata discovered a key pattern: every time the observation tool is upgraded, seemingly chaotic neurons automatically become somewhat clearer.

In 2023, when he worked on OpenAI's first large-scale automated neuron explanation project, the tools of the time could only explain about 2% of neurons. Yet in the modern vision models he tested, 99%–100% of neurons in the final layer could be given quite clear explanations.

The core argument he puts forward in his essay Neural Networks Are Already Clean is: neural networks possess an extremely coherent, elegant internal organizational structure that can be directly read by humans. The problem is not with the neural network at all — it's with our tools for observing it.

"Reversals" of Two Classic Cases

The Truth About the "Cat and Car Neuron"

Feature visualization in the past was essentially using gradient descent algorithms to synthesize the image that most strongly activates a neuron, but the algorithm "takes shortcuts" in pursuit of the highest activation value. Because the ImageNet training set contains an overwhelming number of cat and dog images, the synthesized strongest-activation images would mix in semantic residue, and researchers interpreting along these incidental features naturally misjudged it as an "animal neuron."

Cammarata's team re-observed using two methods — activation path guidance and positive-negative pole cross-validation — and discovered that the neural network was actually classifying by geometry: it detects a V-shaped geometric form.

The "Car and Dog Neuron" Was Also Misread

In InceptionV1's training data, dog images far outnumber car images, so the optimization process falls into the "dog image basin" and struggles to escape, producing synthesized strongest-activation images that are all dogs.

After introducing automated Agent tools for dynamic gradient tracking and ablation experiments, the team found that this group of neurons detects "whether an object is facing the camera head-on and is left-right symmetric." The car's feature signal, once tagged here as "facing forward," continues to flow downstream to deeper layers, ultimately splitting into the true "car classification circuit" and "dog classification circuit."

Recursive Generalization: Neurons Grow Like Tree Branches

If neural networks aren't randomly superposing, where does the "polysemanticity" phenomenon come from? Cammarata offers a new explanation: recursive generalization.

A neuron might initially only recognize rapeseed flowers. Along the "yellow" feature axis, it gradually expands to recognize all yellow objects; along the "circular shape" axis, it extends to various spherical objects; and following the associative path of "creatures in flower fields," it further covers dogs in flower fields, eventually extending to green broccoli.

In language models, this phenomenon is even more dramatic. A neuron originally detecting only the "iy" sound extends during training to recognize the word "diy," and ultimately even the phrase "Do it yourself" — which contains no "iy" sound at all — can activate it.

More critically, there's the extreme stress test. Cammarata compressed 480 neurons in a certain layer of InceptionV1 originally used to detect curves down to 10, a 48-fold compression, and retrained from scratch. According to the superposition hypothesis, the neural network should cram unrelated features into these 10 neurons haphazardly, but the result was the exact opposite — each of these 10 neurons was clearly interpretable, just with broader coverage.

The features "disentangled" by SAEs actually had response widths several times broader than the original neurons. This means that those superposed features claimed to be disentangled from the network's black box are very likely mathematical illusions manufactured by the SAE tool itself.

The Mechanistic Meaning of Recursive Generalization and Feature Axes

Recursive generalization describes the phenomenon in which neurons gradually expand their responses along continuous dimensions of semantic or perceptual features during training. You can think of it as a tree continuously branching from a certain "seed feature" (such as "yellow"): in each round of training, as long as a new sample is close to the existing response along some feature axis, the neuron will be further activated, pushing the response boundary outward. This differs from the traditional assumption that "each neuron corresponds to a fixed concept" — it implies that a neuron's "polysemanticity" is actually a continuous trace left by training dynamics, rather than a compromise in which the network is forced to fold unrelated concepts together. In language models, this phenomenon manifests as layer-by-layer extension from phonemes and word forms to phrase semantics (e.g., from "iy" to "Do it yourself"). Recursive generalization provides a positive explanation for "polysemanticity is not a defect," and also explains why seemingly chaotic neurons reveal a readable internal order after tool upgrades.

Debate and Doubts: What Exactly Are the Two Sides Arguing About?

After the essay was published, Neel Nanda — one of the core founders of superposition theory and head of the mechanistic interpretability team at Google DeepMind — poured cold water on it directly, saying the article was "interesting, but doesn't bring any substantive breakthrough for interpretability."

Neel pointed out that the two sides are fundamentally not arguing about the same "superposition": one is "arbitrary superposition" — the network cramming completely unrelated features into one neuron, which is the version Cammarata refutes; the other is "representational superposition" — large language models using very few neurons to express massive numbers of concepts through linear combinations. The two targets were never in the same place to begin with.

Neel also raised doubts from three dimensions: weak neuron activations store massive combinations of facts and memories, which is precisely the core vehicle of superposition; the paper's most solid evidence all comes from vision models, while language models have information density several orders of magnitude higher; SAEs remain an irreplaceable tool on the residual stream.

Cammarata also acknowledges that his article is merely a "working hypothesis," with several key questions unresolved: the research only covers unimodal vision and language models, and the circuit organization of fusion layers in multimodal models is completely uncovered; language models lack training-process checkpoint data, so patterns can only be inferred from final form; strict causal validation under purely natural conditions remains an open problem.

Why This Matters

Even as just a working hypothesis, it is forcing practitioners across multiple directions to recalibrate their technical directions.

Interpretability teams can directly look at native neurons and circuits, conducting rapid routine observations rather than obsessing over SAE disentanglement surgery; safety alignment teams — if native networks are naturally readable, it means the barrier to understanding AI drops dramatically; architecture design teams — residual streams and LayerNorm, improvements that make models "learn better," unexpectedly also make networks more readable, so future architecture search can add "interpretability" to its metrics; application-layer engineering teams — regulating neurons through behavioral interventions to change model behavior in specific scenarios may go from a paper concept to a product feature.

Ten years ago, we looked at blurry neuron images and concluded that "neural networks are unexplainable." Ten years later, we discover that what was ever chaotic was never the network, but the observation tools and preconceived biases.

One-line Summary

Nick Cammarata proposes that neuron polysemanticity may be a misinterpretation by observation tools, and that neural networks themselves are already clear enough to read.

Paragraph Summary

Nick Cammarata, an early mechanistic interpretability researcher at OpenAI, in his essay Neural Networks Are Already Clean, overturns a classic conclusion he participated in a decade ago: neuron 4e:55 in InceptionV1, thought to encode both "cat faces" and "cars," actually detects a V-shaped geometric form. He argues that the "polysemanticity" evidence widely cited over the past decade likely stems from crude feature visualization tools and preconceived biases. Using activation path guidance, positive-negative pole cross-validation, and automated Agent tools, he found that 99%–100% of neurons in the final layer of modern vision models have clear explanations, and based on this he proposes "recursive generalization" — neurons grow along feature axes like tree branches. He supports neuron interpretability with a 48-fold compression experiment, and questions whether features disentangled by SAEs are mathematical illusions manufactured by the tool itself. Neel Nanda rebuts that the two sides define "superposition" differently, and that vision conclusions are hard to generalize to language models.