System 1 vs System 2 Models: The Fundamental Difference Between One-Shot Answers and Chain-of-Thought Reasoning

When Fast Thinking Meets Slow Thinking

Since new models like JEV (referred to as Jeff in the video) have been released, discussions around System 1 and System 2 models have been heating up. But many people haven't quite grasped the real difference between the two — or when to use which. This article draws on a YouTuber's hands-on analysis to break it all down clearly.

Both concepts come from Daniel Kahneman's Thinking, Fast and Slow. System 1 is autopilot-style intuition — you see a face and instantly know it's angry. System 2 is a step-by-step rational process, like doing multiplication where you have to hold intermediate results in mind and work through it piece by piece.

A Simple Test That Exposes the Difference

The creator designed a clever test: given an org chart for a small company with three separate reporting chains, the facts are deliberately scrambled so you can't just read from top to bottom. The question is simple: starting from Mo, follow the chain upward — who manages whom — and where do you end up?

He fed this question to two models. The first was Jeff, which almost instantly returned the answer "Eli" with no visible reasoning process. The second was Qwen 3.8 with thinking mode enabled, which produced a complete chain of thought and arrived at the correct answer: "Gus."

The key insight: Jeff wasn't just guessing randomly — Eli happens to be three levels above Mo. It moved in the right direction but stopped one step short. That "missing step" is precisely the core divide between System 1 and System 2.

There's a graph-theoretic basis for why this type of "scrambled hierarchy traversal" test is so effective at distinguishing the two model classes. A management chain is fundamentally a directed tree, and tracing k levels upward is equivalent to finding a path of length k in that graph. Research by Merrill & Sabharwal and others in 2023 theoretically proved that for fixed-depth Transformers, determining whether two nodes in a graph are connected by a path longer than a certain threshold is impossible without intermediate outputs (corresponding to the limitations of the complexity class TC⁰). In other words, this isn't a matter of the model being "not smart enough" or "undertrained" — it's a fundamental architectural constraint. A single forward pass has limited computational depth and cannot encode arbitrarily long sequential reasoning. This provides the theoretical backbone for the later experimental finding that "the longer the chain, the wider the gap" — and explains why having the model write out its steps (i.e., unrolling the loop into the output sequence) directly breaks through this limitation.

System 1: A Single Forward Pass — Just One Look

At the model level, System 1 is a single forward pass — the input travels through a fixed stack of layers exactly once. Think of it as "one look at the problem": whatever the model can compute in that single glance is the answer it has to give.

Jeff as a product is built entirely around this philosophy: text and a question go in, the model runs one forward pass, and it returns a probability distribution over whatever answer options you've allowed. The claim of "no hallucination" means it can only pick from the options you provide — it can confidently give the wrong answer, but it can't fabricate options out of thin air.

In theory, a chat model with thinking turned off works the same way. Each token is generated from "one look" at the problem, with no space to reason things through first. So when the author asked Qwen (thinking disabled) the same question, it answered "Tia" — climbing only one level up.

A "forward pass" refers to the complete computation from input to output in a neural network: input tokens flow through an embedding layer and a series of Transformer layers (each containing self-attention and a feed-forward network), ultimately producing a probability distribution over candidate tokens. The entire process is unidirectional and one-shot — no loops, no backtracking. For autoregressive language models, generating each new token requires one independent forward pass. The critical point is that this pass can only "see" the text already in the current context; it cannot internally cache intermediate reasoning that hasn't been written out yet. This is fundamentally different from how human working memory operates: people doing mental arithmetic can "hold" intermediate results in their heads. But for a System 1 model, if intermediate results aren't written into the context, they simply don't exist. Products like Jeff take this further by constraining the output space to a predefined set of labels — eliminating hallucinations, but also permanently preventing the model from using "drafting" as a reasoning aid.

System 2: The Same Layer Stack Running in a Loop

When a problem requires more than one look, that's where System 2 comes in. Importantly, it's not a different brain or a different architecture — it's the same layer stack, running repeatedly in a loop.

The difference is this: at each iteration of the loop, the model writes down one step's result; in the next iteration, it reads that result back and advances further. This is the essence of chain-of-thought (CoT). In the management chain example, in the first iteration the model might write "Mo's manager is Tia"; in the next iteration it reads Tia and looks for Tia's manager. Each step creates a new result that didn't exist before, and the next step builds on it — just like working out 17 × 24 by hand.

The author references an interesting paper: determining whether two nodes in a graph are connected is provably impossible for a Transformer that must answer immediately. The management chain problem is exactly this kind of problem. As soon as you let the model write out its steps, it can answer correctly.

In one sentence: System 1 is taking one look; System 2 is placing that same look inside a loop and repeating it.

The Longer the Chain, the Wider the Gap

The author ran a reproducible experiment: giving Jeff, Qwen (thinking off), and Qwen (thinking on) the same type of problem with progressively longer chains.

With a single step, all three got it right — this is just simple lookup. But as soon as a second step is required, the two "one-look models" start making errors; a few steps further, they're almost entirely wrong. The thinking model, by contrast, stays near-perfect throughout — the only thing that grows is its token count.

It's worth emphasizing: the model hasn't gotten smarter — letting it "think longer" simply means it's taking more steps via tokens.

The Pattern in Jeff's Wrong Answers

One interesting finding: Jeff's wrong answers are mostly not random. Whether the author asked it to go two, three, or four levels up, the most common answer was almost always "the person three levels up." It walks up the correct chain and then, almost every time, runs out at the same point.

The author's interpretation: this is what "one look" looks like from the outside — it can carry you this far, and no further. Since Jeff's internals aren't available for inspection, we can't say definitively why the limit falls exactly there.

System 1 Isn't Weaker — It's Just for Different Tasks

A key conclusion: this doesn't mean System 1 is inferior. As long as the answer is visible in one look, it's the right tool for the job. Questions like "Is this ticket about billing?" "Is it urgent?" "Does it mention a refund?" — you can use a one-look model to ask dozens of these questions about a simple input all at once. That's exactly what Jeff is genuinely good at.

Where it fails is when you hand it a question that secretly requires multiple steps. Kahneman calls this "substitution": System 1 quietly answers an easier question instead. When the author asked Jeff "Is this email a phishing attempt?" on a difficult dataset, its accuracy was roughly coin-flip level — because it was actually answering "Does this look like a phishing email?" But when he decomposed the same question into sub-questions about specific features and combined the answers in code, accuracy jumped significantly.

Orchestrating the Two Systems with Confidence Scores

Jeff also returns a confidence score with each answer, which makes it considerably more valuable. On the chain problem, when the depth exceeds what can be handled in one look, its confidence drops along with its accuracy — which maps perfectly to Kahneman's description: System 1 answers first, and only when something "feels off" does System 2 kick in.

This enables a cooperative system design: use the one-look model to answer everything while keeping confidence scores, then route any task that falls below a threshold to a thinking model. In an agentic architecture, this might look like System 1 handling tool calls, with uncertain calls escalated to System 2 for more comprehensive judgment. The author believes most future systems will be designed this way.

But there's an important caveat: don't blindly trust confidence scores. They are scaled probabilities, not genuine measures of how confident the model truly is. Any real deployment should tune this threshold against your target data before going live.

The reliability of confidence scores in production is a frequently overlooked trap. The probability values output by language models are softmax-normalized, reflecting relative preference rather than calibrated true probability — a confidence score of 0.9 does not mean the model is 90% likely to be correct. Miscalibration is especially common in models fine-tuned with RLHF or instruction tuning, since the training process systematically inflates probabilities for high-frequency answers. This is why the article recommends "tuning the threshold" against your target dataset before deployment — which is essentially performing confidence calibration: measuring "when the confidence is x, what is the actual accuracy?" on a validation set, then mapping raw scores to more trustworthy probability ranges to set your routing threshold. Platt scaling and temperature scaling are common post-processing calibration methods with minimal computational overhead, well-suited as standard steps in a System 1/2 cooperative architecture.

How to Choose: Count the Steps

The simplest heuristic is to count how many steps the model needs to reach an answer:

• Answer is visible in one look → System 1 (e.g., Jeff)
• Requires creating intermediate results that don't yet exist, built up across multiple steps → System 2 (thinking model)
• Or: decompose the problem yourself into one-look-answerable chunks and combine them in code

The author teased that the next step is to build this thinking into an agent: a thinking model handles planning, while Jeff handles the hundreds or thousands of small decisions in between. The code for the chain test is also in the video description — readers who are curious can try it themselves and see how far they can push a System 1 model before it breaks down.