Is Chain-of-Thought Actually Reasoning? Causal Measurements Reveal the Truth About CoT Faithfulness

The Core Assumption Behind CoT Monitoring Is Under Scrutiny

The chain-of-thought (CoT) capability of large language models is often seen as a critical lever for model interpretability and safety monitoring. Underlying this is a default assumption: the reasoning steps a model writes out faithfully reflect the internal computation that produced the answer. In other words, if a model says "because A leads to B, therefore the answer is C," we assume that text represents the actual basis for arriving at C.

But does this assumption hold up? A newly published arXiv paper (arXiv:2609.27038) offers one of the most rigorous causal-level answers to this question to date. The central finding: the reasoning steps a model states are indeed causally load-bearing to a substantial degree — but traditional behavioral tests systematically overestimate this faithfulness, especially on simpler tasks where the reasoning appears most fluent.

From "Behavioral Observation" to "Activation Intervention"

Previous methods for measuring CoT faithfulness have been largely behavioral: researchers edit the reasoning text the model produces, then observe whether the final answer changes. If modifying a reasoning step causes the answer to shift, that step is deemed "useful." This approach is intuitive but only captures input-output correlations — it cannot establish causation.

The key innovation in this paper is pushing measurement down to the activation level, directly performing a causal audit on the model's own generated reasoning. The approach works by replacing the residual stream of the model within the token span of a stated intermediate reasoning step with the corresponding activation values from a "counterfactual run."

Unlike prior causal audits that measured "performance degradation," each intervention here carries a known prediction target — every patch should theoretically flip the answer to a specific counterfactual entity that can be derived by construction. This gives the experiment a clear reference baseline rather than vaguely observing "how much worse it got."

Synthetic Multi-Hop Lookup Tasks

To keep causal chains clear and controllable, the study uses synthetic multi-hop lookup tasks ranging from 2 to 6 hops. Multi-hop tasks are ideal because the reasoning path is explicit and intermediate entities are enumerable, making it easy to precisely specify counterfactual targets and verify the effects of interventions.

The residual stream is the core information-passing mechanism in the Transformer architecture. In a standard Transformer, attention heads and feed-forward networks at each layer don't directly overwrite the representation — instead, they add their outputs to a backbone vector that runs through the entire network. This backbone is the residual stream. Because each layer's modifications are applied additively, the state of the residual stream at any position theoretically contains all information written by every preceding layer. Activation patching exploits this structure: at a specific token position and layer, the researcher replaces the current residual stream vector with the corresponding vector from a separate "counterfactual" forward pass, thereby injecting different information content in a targeted way while leaving activations at all other positions and layers unchanged. This surgical intervention allows researchers to ask, in a near-controlled-experiment fashion: how much of the model's computation in generating an answer actually depends on a stated reasoning segment, versus other sources?

Multi-hop lookup tasks typically take the form of a series of chained key-value queries — for example: given "entity A's attribute is B," "entity B's attribute is C," and so on, the model must start from an initial entity and hop through the chain until it reaches the target entity at hop N. These tasks are well-suited for causal auditing because the reasoning path is entirely transparent — every intermediate entity is an enumerable ground truth. Researchers can precisely construct counterfactuals: simply replace the fact at one hop with a version pointing to a different entity, and you can derive exactly what the final answer should become if the model truly used that step. By contrast, natural language reasoning tasks have semantically ambiguous intermediate steps and hard-to-specify counterfactual targets, which introduces measurement noise. The graduated design spanning 2 to 6 hops offers an additional advantage: as hop count increases and reasoning chains grow longer, researchers can systematically examine how well models maintain faithfulness under increasing cognitive load.

Qwen3-4B: ~77% of Steps Are Truly "Load-Bearing"

On the Qwen3-4B model, the study finds that at the most sensitive intermediate layer, 76.9% (±2.8%) of stated reasoning steps are causally load-bearing (CLB).

This number requires a few reference points to appreciate its significance:

• The null baseline at random positions is only 11.3% — confirming the intervention effects are not random noise.
• Directly patching the underlying prompt facts achieves an upper bound of 83%.

This means the model's stated reasoning steps carry approximately 96% (76.9% relative to 83%) of the achievable causal effect. This is a remarkably high proportion, indicating that for a model of this scale, the written reasoning is not purely "post-hoc rationalization" — it genuinely participates in generating the answer.

Behavioral Tests Overestimate Faithfulness

The more thought-provoking finding is the gap between behavioral and causal testing. On the exact same questions, standard behavioral tests yield a faithfulness score of 88.2% — 11.4 percentage points higher than the causal measurement of 76.9%.

This gap is highly statistically significant (item-matched pairing, 111:14 discordant pairs, p < 1e-15). More concerning: on the simplest questions, this overestimation reaches as high as 20 percentage points.

This means that precisely when a model's reasoning appears most fluent and most "convincing" is when behavioral tests are most easily misled. Surface-level coherence masks the misalignment between internal computation and the stated text. For scenarios relying on CoT monitoring for AI safety auditing, this is a blind spot that cannot be ignored.

The mechanism behind this overestimation deserves further explanation. Behavioral tests re-feed the model with edited reasoning text, which fundamentally changes the entire context — not just the internal activations. When reasoning steps are altered and the answer appears to change accordingly, this change may arise from multiple confounding factors: the edit introduces grammatical or semantic anomalies that trigger the model's robustness responses to "strange inputs"; or the modified text happens to match certain training patterns at a surface level, directly activating different answer modes. This confounding effect is especially pronounced on simple tasks — answers often have high prior probabilities, so behavioral perturbations are easily over-interpreted as "causal responses." Activation intervention bypasses these surface-level confounds and directly tests whether internal representations participate in the computation, yielding a more conservative and accurate estimate.

Faithfulness Correlates Strongly With Model Capability

The research also reveals a capability dimension to faithfulness. The smaller Qwen3-1.7B has an overall causal faithfulness of only 54.8%, far below the 4B version.

More critically, the decay pattern with reasoning depth is striking:

• Qwen3-1.7B achieves 68% faithfulness on 2-hop tasks, but this plummets to 30% on 6-hop tasks.
• Qwen3-4B, by contrast, remains relatively stable across different hop counts.

In other words, less capable models increasingly produce reasoning that is "in name only" as reasoning chains grow longer — the model may write out elaborate reasoning on paper, but the actual computation has already decoupled from the text. This suggests CoT faithfulness is not a fixed property, but one that varies dynamically with model scale and task complexity.

Implications for AI Interpretability and Safety

The value of this research lies not in dismissing CoT, but in calibrating how much we should trust it. Several practical takeaways are worth highlighting:

First, behavioral tests require careful interpretation. Relying solely on editing text and observing answer changes will systematically overestimate the true causal role of reasoning, especially on simple tasks. Causal intervention at the activation level provides a more reliable measuring stick.

Second, model scale affects trustworthiness. Smaller models' reasoning chains are more prone to "distortion" as depth increases; CoT monitoring for small models warrants heightened vigilance.

Third, fluency does not equal faithfulness. The more smoothly reasoning flows, the more likely it is to give the false impression that it reflects genuine computation. This poses a challenge for automated safety auditing based on CoT — the uncertain situations most in need of monitoring may be precisely those where the verbal expression appears most confident.

As chain-of-thought reasoning becomes a standard capability for frontier models, verifying whether "what the model says" matches "what the model does" will be a long-term challenge for interpretability research. This paper provides a reproducible causal measurement framework, laying the methodological groundwork for future work.