One Tweet, One Warning: The AI Community's Misuse of 'System 2' Thinking

One Tweet That Got People Thinking

A joke tweet made the rounds recently among AI practitioners: "A person who has only read Thinking, Fast and Slow says while training his machine learning model — 'I can really feel the System 2 vibes here.'"

What reads as a throwaway gag actually hits on a very real phenomenon in the AI community: borrowing concepts from psychology and cognitive science and loosely applying them to machine learning models, often without any rigorous grounding.

What 'System 1' and 'System 2' Actually Mean

Thinking, Fast and Slow is the defining work of Nobel laureate Daniel Kahneman. In it, he describes two modes of human thought:

• System 1: Fast, automatic, intuitive thinking that requires little effort — recognizing a face, understanding a simple sentence.
• System 2: Slow, deliberate, effortful thinking that demands focus — solving a complex equation, working through logical reasoning.

This framework is genuinely intuitive, which is exactly why it became a popular lens for understanding cognition — and why it has been borrowed so freely in AI discourse, especially when discussing the "reasoning capabilities" of large language models.

Why the AI World Loves This Analogy

With the rise of reasoning models in recent years, "System 2 thinking" has become a fashionable label for a model's capacity for "deep thought." When a model generates a chain-of-thought before arriving at an answer, it's almost reflexively described as "System 2-like" behavior.

The appeal of the analogy is understandable. It offers a ready-made, universally accessible framework that translates complex model behavior into everyday language. Investors, journalists, and researchers alike have embraced it because it lowers the communication barrier.

But that's precisely the problem. When someone has "only read" the one book and rushes to interpret their model through that lens, they often end up with a superficial mapping — using a psychological metaphor to paper over a genuine lack of understanding of how the model actually works.

Chain-of-Thought (CoT) prompting was systematically introduced by a Google research team around 2022. The core idea is to have a model generate a series of intermediate reasoning steps before producing a final answer. Experiments showed this approach significantly improved large language model accuracy on tasks like math word problems and logical inference. Models like OpenAI's o1/o3 series and DeepSeek^Ad-R1 take this further, using reinforcement learning to encourage models to autonomously generate longer "chains of thought" during inference — including self-correction of intermediate steps. It's this outward appearance of "thinking before answering" that makes observers feel it closely resembles the deliberate, effortful quality of Kahneman's System 2. But the resemblance is behavioral. There is no underlying mechanism in these models for "allocating attentional resources" or "suppressing intuitive impulses."

Where the Analogy Breaks Down

Applying human cognitive frameworks to machine learning models is, at its core, a form of anthropomorphism. It works well for communication; it can mislead badly for technical judgment.

The "reasoning" of a large language model and human System 2 thinking differ fundamentally in mechanism. A model's chain-of-thought is token generation driven by statistical patterns — it is not "deliberate focus" or "logical deduction" in any meaningful sense. Treating the model's intermediate output steps as equivalent to the brain's slow thinking risks overstating the model's cognitive capabilities and can distort how we design and evaluate models.

This is exactly what the tweet is satirizing: genuine experts carefully define the limits of an analogy, while someone armed with a single popular concept is more likely to apply it everywhere and stretch it far beyond what it can support.

Anthropomorphism has a long history in AI discourse, but its harms often extend beyond mere semantics. Cognitive scientist Gary Marcus and philosopher John Searle, among others, have long argued that when we describe models using human-exclusive terms like "understanding," "reasoning," or "consciousness," we implicitly assume the model has the internal foundations for these capacities — which in turn shapes research priorities, regulatory approaches, and public perception of AI risk. In the specific case of the dual-systems framework: System 2's defining features include limited working memory capacity, the need to consume cognitive resources, and susceptibility to fatigue — all constraints tightly bound to biological neural systems. Large language models perform the same depth of forward pass at every token generation step. There is no corresponding mechanism for "fatigue" or "resource depletion." So even when a model's output looks "slow and deliberate," the underlying computational logic remains structurally distinct from human slow thinking.

What This Means for Practitioners

This little corner of internet humor actually points to a broader methodological question in AI: how do we borrow concepts from other disciplines without losing rigor?

There's nothing wrong with drawing inspiration from cognitive science frameworks for model design — Kahneman's dual-systems theory has genuinely sparked valuable lines of research. But the key is this: an analogy should be a starting point, not a conclusion. When we say a model exhibits "System 2" characteristics, we should be able to articulate both where that correspondence holds and where it breaks down.

For content creators, investors, and everyday users: when you encounter words like "System 2," "intuition," or "consciousness" being used to describe AI, it's worth pausing to ask — is this a precise technical description, or a compelling marketing narrative?

Maintaining a critical distance from popular concepts may be the most practical lesson this tweet has to offer.