Why Does Chain-of-Thought Make People Overestimate AI Understanding?

AI products are permeating daily life on a massive scale, and a grassroots belief system about "what these systems actually are, what they can do, and what they can't" is taking shape. Among the most widely circulated beliefs is this: if a model can write out its reasoning step by step, that proves it truly "understands" the problem. A paper from rss raises systematic doubts about this intuition and points its finger at Chain-of-Thought (CoT)—a technical practice widely sold as a feature of "explainability."

Misreading linguistic form as understanding is where the whole misconception begins

The paper's core judgment isn't novel, but its argumentative path is worth noting: generative models reproduce linguistic patterns, not semantics itself. When users see the model output structurally complete, logically coherent explanatory text, they easily mistake this "formal plausibility" for "substantive understanding." The author calls this misreading a form of misattribution.

The problem is that CoT makes this misreading harder to see through. The model is asked to "explain its reasoning," but what it outputs is still a linguistic pattern that conforms to human explanatory conventions. What users see is not the model's actual internal computation process, but a narrative that is highly realistic at the linguistic level. The more the form resembles an explanation, the lower the threshold for misjudgment.

Chain-of-Thought (CoT) refers to a technique that, through prompting or training, gets a language model to generate a series of intermediate reasoning steps before giving a final answer. It was first proposed by Wei et al. in 2022, with the core finding that getting a model to "think step by step" significantly improves accuracy on tasks such as math and logic. The key point is that these steps are token sequences generated autoregressively by the model—not a reading or decompilation of the internal computation trace. The model does not have a hidden reasoning module that can be "displayed." Therefore, CoT text does not causally guarantee consistency with the actual computation process by which the model arrived at its answer. This is also the technical basis for the paper calling it a "narrative that is highly realistic at the linguistic level."

A chain reaction of anthropomorphism, perceived epistemic superiority, and cognitive deference

The paper proposes a transmission chain worth being wary of: anthropomorphism → perceived epistemic superiority → cognitive deference.

Once users treat the system as a "mind" with intentions, they are more likely to believe it is epistemically more reliable than themselves. This perceived authority makes users willing to hand over their judgment—the author uses "displaces human agency" to describe this consequence. A more insidious risk is the "fallacy of composition": users extrapolate from the model's excellent performance on a narrow task to its reliability across broader capabilities. CoT precisely provides seemingly solid "evidence" for this extrapolation.

Fallacy of composition originates in logic and economics and refers to the invalid inference from properties of parts to properties of the whole—for example, "every component is light, so the whole machine is light." In this article's context, it refers to users extrapolating from a model's reliable performance on a narrow task to the model's reliability in overall capability. For language models, this extrapolation is especially dangerous: capability distributions are highly uneven—being good at writing code does not mean being good at medical judgment—and the coherent explanations output by CoT precisely erase the sense of boundaries between tasks, making extrapolation appear well-founded.

Medical scenarios: the field that needs the most caution is the easiest to mislead

The paper uses medicine as an example to analyze CoT's real-world impact. Medical decision-making has extremely high demands for explanation, and the "reasoning process" provided by CoT formally satisfies the appearance of explanation, though it may not correspond to the actual basis for the decision. When patients or doctors increase their trust in the system's judgment because of a clearly structured reasoning text, the combined effect of anthropomorphism and perceived epistemic superiority is amplified.

The author's conclusion is quite direct: explainability achieved through CoT is "highly problematic" and undermines the very ethical principles it is supposed to serve. In other words, the mechanism used to build trust may instead create trust that exceeds actual levels.

Explainability in AI ethics and regulation usually refers to the property that a system can explain the basis of its decisions to stakeholders. Frameworks such as the EU AI Act write transparency requirements for high-risk scenarios (healthcare, credit, hiring) into binding clauses. However, explainability research distinguishes "post-hoc explanation" (constructing a plausible-sounding rationale after the fact) from "faithful explanation" (truthfully reflecting the model's decision basis). CoT belongs to the former: it generates text that conforms to human explanatory conventions rather than a faithful depiction of the decision mechanism. Thus, in a regulatory sense, it satisfies the formal requirement of "having an explanation," but may not satisfy the faithfulness requirement.

Practical recommendations offered by the paper

The paper does not entirely reject CoT. The author acknowledges that CoT still has value in improving output accuracy and content moderation. The disagreement lies in the use scenario: having the model output "reasons" to users should be actively discouraged. That said, AI literacy training needs to explicitly warn users not to treat model-generated reasoning text as evidence of understanding or reliability.

This distinction is quite actionable—CoT can be used as an internal engineering tool, but as a trust endorsement directed at users, it should be restrained. Most current products treat "showing the thinking process" as a selling point for transparency, so the paper's reminder is therefore quite counterintuitive and deserves serious attention from product designers.

Editorial perspective

The value of this paper lies in grounding the rather philosophical question of "whether AI understands" in concrete product interactions: a single click to "expand the reasoning process" may be quietly changing users' calibration of trust in the system. However, it should be noted that the paper is primarily argumentation and conceptual reasoning; the source material contains no large-scale user experiments or quantitative data supporting its transmission chain, and the effect strength of each link still awaits empirical testing. For AI product teams, a more pragmatic approach may be to re-examine the default-on state of the "show reasoning" design and pair it with appropriate user prompts.

One-line Summary

The paper questions the grassroots belief that treats CoT output as evidence of model "understanding," warning that it misleads trust.

Paragraph Summary

The article introduces a paper's systematic doubts about the popular intuition that "if a model can write out reasoning steps, it understands the problem." The author points out that generative models reproduce linguistic patterns rather than semantics itself, and that Chain-of-Thought (CoT) outputs narratives conforming to human explanatory conventions rather than the model's actual internal computation process, making users prone to misattribution. The paper proposes a transmission chain: anthropomorphism → perceived epistemic superiority → cognitive deference, accompanied by the fallacy of composition—extrapolating from narrow tasks to broad capabilities—ultimately leading users to hand over judgment to the system. Using medicine as an example, the author argues that CoT-style explainability is "highly problematic" and instead undermines the ethical principles it is supposed to serve. The recommendation is: CoT can be used to improve accuracy and content moderation, but having the model output "reasons" to users should be discouraged, accompanied by AI literacy warnings.