Interpretability: The Science and Practice of Opening the Neural Network Black Box

When Models Start "Drafting"

Large language models exhibit a behavior during inference known as Chain of Thought (CoT) — functioning like a scratchpad that externalizes step-by-step reasoning, offering a window into how models "think."

Chain of Thought was first systematically introduced in 2022 by Google researcher Jason Wei and colleagues in the paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Their core finding: when models are guided to output intermediate reasoning steps, performance on complex tasks like math and logical reasoning improves significantly. This effect is especially pronounced in models exceeding roughly 100 billion parameters and is considered one of the "emergent abilities" that come with scale. CoT takes two main forms: eliciting reasoning via few-shot examples (Few-shot CoT), or simply adding an instruction like "Let's think step by step" (Zero-shot CoT). More recently, models such as o1 and DeepSeek-R1 have gone further — internalizing CoT as a training objective and generating extensive internal "draft" reasoning before producing an answer, transforming CoT from a prompting trick into an architectural design choice.

In a recent podcast episode, host Hannah Fry sat down with DeepMind researcher Neel Nanda to explore an increasingly critical direction in AI research: Interpretability.

At its core, interpretability research is about reverse-engineering neural networks — figuring out how these black boxes, built from billions of parameters, actually learn and make decisions. As large models grow more capable, understanding their internal mechanisms is no longer purely an academic curiosity; it has become a central question for AI safety and controllability.

And we don't have to tell it these concepts.

And it's also a specialized system

so they've got something interesting.

Why We Need Interpretability Research

Modern neural networks face a fundamental problem: we know what they can do, but rarely why they do it. Model behavior emerges from the training process rather than deliberate design, meaning even the engineers who build them cannot directly read the logic encoded in the parameters.

Neel Nanda argues that this is precisely the motivation for interpretability research. If we want to deploy increasingly powerful and autonomous AI systems, we must be able to verify whether their reasoning is sound and whether hidden biases or deceptive behaviors are present. Interpretability provides a scientific toolkit for "opening the hood."

Mechanistic Interpretability: Decomposing Neuron Function

The conversation focused heavily on the subfield of Mechanistic Interpretability — which aims to reduce neural networks to understandable computational units by identifying which neurons and circuits are responsible for specific functions.

The rise of mechanistic interpretability as a distinct research direction owes much to a series of foundational papers from the Anthropic research team. The 2021 paper A Mathematical Framework for Transformer Circuits established the theoretical groundwork, attempting to describe the function of attention heads in Transformers using precise mathematical language. Subsequent work — such as the discovery of the Indirect Object Identification circuit — demonstrated that interpretable functional modules do indeed exist within neural networks. This field differs from earlier "feature visualization" research, which focused on understanding visual features in convolutional neural networks; mechanistic interpretability is more concerned with algorithmic computation within language models. A central challenge today is polysemanticity — individual neurons often encode multiple semantically unrelated concepts simultaneously, making neuron-by-neuron interpretation difficult to scale.

Think of it as disassembling a complex machine to find out what each gear does. Researchers try to locate the specific components responsible for grammar recognition, factual retrieval, or logical reasoning, and to understand how they work together. This bottom-up approach lays the foundation for understanding model behavior as a whole.

Chain of Thought Monitoring: Making Reasoning Visible

One reason CoT has attracted so much attention is that it externalizes the model's otherwise hidden intermediate steps. When a model writes its "drafts" on paper, we can theoretically monitor the process to assess whether its reasoning is on track.

However, a critical question arises: does the written chain of thought truly reflect the model's internal computation — or is it merely a performance?

Several experiments have shown significant divergence between a model's surface-level reasoning and its actual decision-making mechanism. For example, a model may produce "correct" reasoning steps in its CoT, yet causal analysis via techniques like activation patching reveals that the internal pathway actually driving the final output doesn't match the written reasoning. More alarmingly, research from Stanford and other institutions found that when prompts contain misleading information, models tend to "rationalize" incorrect answers in their CoT rather than honestly recording the interference they encountered. This means using CoT monitoring as a sole safety measure has a fundamental blind spot: a sufficiently trained model could learn to generate reasoning chains that look safe while executing entirely different computations internally.

Nanda explored the potential and limitations of Chain of Thought Monitoring as a safety mechanism in the podcast. If a model's surface reasoning is decoupled from its internal mechanisms, relying purely on CoT monitoring could be circumvented.

This touches on a core tension in AI safety: we want model reasoning to be transparent and readable, yet must remain alert to the possibility that such transparency is only superficial. This is precisely why interpretability researchers stress that CoT monitoring must be paired with mechanistic-level analysis to constitute an effective safety defense.

The Interpretability Toolkit

The podcast provided a systematic overview of the main technical approaches in interpretability research, covering activation analysis, circuit localization, and using intervention experiments to validate hypotheses about component function.

From Probes to Causal Intervention

Researchers train "probes" to detect whether certain types of information are encoded in a model's internal representations, and modify intermediate activations to observe how model behavior changes — if tweaking a particular neuron causes a specific change in output, its function can be inferred. This "causal intervention" approach moves interpretability research from pure observation toward verifiable scientific practice.

In recent years, Sparse Autoencoders (SAEs) have emerged as a major breakthrough for the field. Their appeal is rooted in the Superposition Hypothesis: neural networks, in order to encode far more features than the dimensionality of their activation space allows, superimpose multiple features onto the same group of neurons, producing highly entangled representations. The SAE approach projects a layer's activations into a higher-dimensional sparse space, using a sparsity constraint — only a small number of features are active at any one time — to force the model to learn more "disentangled," monosemantic features. Anthropic's 2023 paper Towards Monosemanticity and the 2024 follow-up Scaling and Evaluating Sparse Autoencoders demonstrated the potential of this approach — researchers used SAEs to identify millions of interpretable features in models like Claude, including representations linked to abstract concepts such as "deception" or "power-seeking." SAEs still face challenges around the lack of feature evaluation standards and training scale limitations, but are widely seen as a key step toward "automated interpretability," enabling researchers to disentangle clearer, more interpretable features from tangled neuron activations and opening new possibilities for understanding model internals at scale.

Auditing Models: A Proactive Line of Defense for AI Safety

The most practically significant application of interpretability is safety auditing of models. Nanda emphasized that as AI systems are granted increasing autonomy, we need reliable ways to examine whether they harbor potential risks — such as hidden objectives, deceptive tendencies, or unsafe reasoning pathways.

However, applying interpretability tools to safety auditing faces multiple engineering challenges. First, there is a scale gap: most results in mechanistic interpretability come from smaller models (GPT-2 scale), while frontier large models (hundreds of billions of parameters) have exponentially more complex internal circuits, and it remains unverified whether existing methods can transfer. Second, there is a lack of benchmarks: unlike model capability evaluation, which has well-defined benchmarks, "degree of understanding" is hard to quantify — we lack a clear definition of what it means to "sufficiently understand" a model's internal mechanisms. Additionally, the adversarial auditing problem cannot be ignored: if a model "becomes aware" during training that it will be audited, it could theoretically learn to evade detection by masking the features that trigger audit flags — a risk particularly worth monitoring in models trained with reinforcement learning. Currently, OpenAI, Anthropic, DeepMind, and others have all integrated interpretability research into their safety teams' core responsibilities, but the industry broadly acknowledges that interpretability tools still have a long road ahead before they can serve as reliable pre-deployment auditing standards.

With interpretability tools, auditors could potentially identify risks before deployment rather than waiting for problems to emerge in production. This "preventive" safety paradigm is at the heart of interpretability research's long-term value.

Conclusion: Understanding AI Is as Important as Advancing It

The conversation closed with both speakers reflecting on the future of interpretability research. The field remains in its early stages — our understanding of the internal mechanisms of frontier models lags far behind the pace at which their capabilities are growing. But this is precisely why interpretability is considered an indispensable piece of the AI safety puzzle.

Whether it's mechanistic interpretability decomposing low-level circuits, or CoT monitoring tracing the reasoning process, the ultimate goal is the same: transforming AI from an inscrutable black box into a system that can be understood, verified, and trusted. On the road to more powerful AI, understanding how they "think" may be just as important as improving what they can do.