When Is a Reasoning Model Worth Paying For? Three Criteria to Guide Your Decision

Reasoning models consume extra tokens for internal thinking before delivering an answer, which brings accuracy gains along with latency and cost. The question is: not every request is worth that overhead. In Episode 54 of the Chain of Thought podcast, Dan Klein offered a straightforward decision framework — a reasoning model is only worth paying for when you can articulate what the extra thinking is buying you.

The Core Difference Between Reasoning Models and Standard Models

A standard model produces an answer in a single forward pass. A reasoning model is trained to first run an internal chain of thought, exploring, checking, and correcting before committing to an answer. This is what test-time compute means: the model parameters stay the same, but more computation is invested at answer time. On math, code, and multi-step logic tasks, this thinking can significantly improve accuracy. DeepSeek^Ad's work has demonstrated that by using reinforcement learning to automatically generate step-by-step reasoning data at scale, a model can develop stronger reasoning ability through heavier inference.

Before reasoning models became the default option, chain-of-thought prompting was the primary technique. A 2022 paper by Wei et al. at Google Brain showed that asking a model to produce steps in the prompt can improve accuracy on math, logic, and planning tasks. Few-shot chain of thought provides completed examples, while zero-shot chain of thought simply adds "let's think step by step" — cheaper, and often sufficient on capable enough models. Self-consistency samples multiple chains and keeps the majority answer, trading more tokens for further gains on hard problems. On reasoning models, these prompting techniques matter less because long chains are already built in.

But a fluent reasoning chain can be wrong. AI hallucination refers to false or fabricated content stated as fact, and it stems from the mechanisms by which these models generate text. A plausible-looking reasoning chain can coexist with an incorrect result — the presence of a chain does not make the output trustworthy.

test-time compute refers to the extra computation a model consumes during inference (rather than training). The traditional view held that a model's capability is fixed after training, and improvements could only come from larger parameters or more data. Test-time compute changes that assumption: with the same model weights, generating longer chains of thought, sampling multiple candidate paths, or performing self-verification at answer time can yield higher accuracy on hard problems. Reasoning models like OpenAI o1 and DeepSeek-R1 have turned this into a training objective — using reinforcement learning to reward reasoning trajectories that lead to correct answers, teaching the model to "think longer." The cost is that latency and token consumption grow linearly with thinking length, so it is essentially a trade of compute for accuracy, not a free improvement.

Self-consistency is a decoding strategy proposed by Wang et al. in 2022: sample multiple independent reasoning chains for the same question at higher temperature, then take a majority vote on the final answer. The intuition is that correct answers tend to be reached by multiple different paths, while random errors scatter. Compared to a single chain, self-consistency typically yields a few percentage points of improvement on math and commonsense reasoning, but at the cost of multiplied tokens and latency. In the era of reasoning models, its positioning has therefore become a "middle tier" — more expensive than a standard single-pass answer, but cheaper than a full reasoning model, suitable for trying first in scenarios where a checker exists and multiple samples are affordable.

Three Checks Before You Pay

Before deciding to route a request to a reasoning model, pass it through three checks in order. If any one fails, the request stays on the standard model.

First, can you tell which answer is correct? Dan Klein's simple picture of a reasoning model is that it tries multiple strategies and keeps the one that works. This helps in math: once the correct strategy is found, the proof goes through. You need a concrete basis for judging these attempts — unit tests, schemas, a calculator, or known labels. If you cannot distinguish a good answer from a fluent one, a longer chain just gives you a longer wrong answer.

Second, does the latency budget allow a slow answer? More test-time compute means slower responses. A chat reply a user is waiting for may not have room for a long internal chain. Batch tasks, CI checks, or overnight reports usually can.

Third, is the cost of thinking tokens less than the cost of a wrong answer? Klein's warning about when to skip this overhead is direct:

Humans go further. A pebble flies at your face, you just blink — you don't stop and think "if the stone hits my eye that's bad, I could turn my head, I could close my eyes." You just blink. As humans, we use different amounts of computation and metacomputation depending on the situation. Reasoning is a very particular use of test-time compute, and it works for some problems and is absolutely not the right solution for others.

— Dan Klein, co-founder and CTO of Scaled Cognition, Chain of Thought Episode 54

He also proposed that once a task has been done a few times, planning should become memory. Rather than solving from scratch on every request, cache the known procedure or just call the standard model.

There may be a cheaper intermediate step between the two models that is worth measuring. On a capable enough standard model, try zero-shot chain of thought first; if the checker still fails and you can afford multiple samples, then try self-consistency. Only enable the reasoning model for the slice of requests that still fail those steps and still pass all three checks.

Checker / verifier here refers to any mechanism that can automatically determine whether an answer is correct: unit tests, JSON schema validation, calculator verification, known database labels, or exact matching against a reference answer. It is the foundation of the entire routing framework, because the value of a reasoning model comes from "trying multiple strategies and keeping the one that works," and "works" must be defined by an external signal — the model cannot reliably judge this itself. Without a checker, you can only compare two fluent answers, and fluency is exactly what hallucination is best at faking. This also explains why math, code, and structured data extraction suit reasoning models — they come with checkers built in; whereas open-ended writing, chitchat, and subjective recommendations do not, because correctness cannot be automatically verified.

How to Measure the Difference on Your Own Tasks

Public benchmarks are only a shortlist. The real deployment decision comes from tasks you control. An AI benchmark is a fixed test plus a shared scoring method — MMLU for knowledge, HumanEval for code, for example. But a leaderboard-leading model can still fail on your data, your format, and your edge cases, and benchmark questions may leak into training data, inflating scores. Use public scores to pick candidate models; use your own scoring to decide routing.

Dan Klein pointed out that bad metrics let you convince yourself you are making progress that isn't there. He contrasted two kinds of systems: one that is correct in most scenarios but errs often enough to break customer actions like booking a flight; another that covers fewer scenarios but is correct every time. The latter is what you can ship. He also argued that every team must decide what it needs and check whether the metric captures that, because it is easy to believe a system will meet the need when in fact it won't.

Draw a small set from real traffic: hard multi-step tasks, simple lookups, and routine cases already solved. Run both the standard model and the reasoning model on each item, and record whether the checker passes, how many tokens were returned, and whether latency is within budget. The reasoning model should only get calls on request categories where the accuracy gain justifies the extra tokens and slower response. On lookup-type tasks, expect the table to show overkill, keeping those calls on the standard route.

If you cannot build a checker, exclude those items from automated comparison and sample manually instead. Treat confident reasoning chains as untrusted text.

Routing Sketch and Practical Checklist

Below is a sketch of a routing rule plus a private scorecard. Note that this is just the author's illustrative example — not a design adopted by any guest or company, and it calls no vendor SDK.

# Sketch only. A routing rule plus a private scorecard.
def route(task, budgets):
    if not task.checkable:
        return "standard"
    if budgets.latency_ms < task.reasoning_p95_ms:
        return "standard"
    if task.expected_thinking_tokens > budgets.max_thinking_tokens:
        return "standard"
    return "reasoning"

def scorecard(items, run_standard, run_reasoning):
    rows = []
    for item in items:
        a, b = run_standard(item), run_reasoning(item)
        rows.append({
            "id": item.id,
            "standard_ok": item.check(a.text),
            "reasoning_ok": item.check(b.text),
            "token_delta": b.tokens - a.tokens,
            "latency_delta_ms": b.latency_ms - a.latency_ms,
        })
    return rows

Read those rows before pointing route at live traffic. When reasoning_ok is true, standard_ok is false, and both deltas are within budget, that category can go to the reasoning model. When both checks pass, keep the standard model. When item.check cannot run, that row doesn't belong in this loop.

Practical checklist:

• Pay for a reasoning model only when a checker exists, the latency budget covers a slow answer, and the extra thinking-token cost is less than the cost of a wrong answer.
• Before making a reasoning model the default, try zero-shot chain of thought on the standard model first, then self-consistency.
• Use public benchmarks to filter, then score both models on your own items: pass rate, tokens, latency.
• Keep lookups, routine work, and uncheckable text on the standard model.
• Re-score when the task mix changes. A model that thinks brilliantly on math may be the wrong expense on the rest of your traffic.

The core logic of this framework is: the extra computation of a reasoning model is an investment, and the return is improved accuracy. That investment only holds when you can verify the return and it is worth the latency and token cost. For most production systems, this means hybrid routing rather than a one-size-fits-all approach — reserving reasoning models for the slice of requests that genuinely need them.