Is Cranking Up Reasoning Effort Worth It? 1,800 Test Runs Across 4 Models Give the Answer

Almost every mainstream model API has a parameter called reasoning_effort, thinking, or reasoning. Developers set it every day, but almost no one has measured its real effect. One developer built a benchmark called Reasoning Dial on Kaggle Benchmarks, with the only variable being that knob. After running 1,800 scored calls across 4 models, the conclusion isn't flattering: it fixed exactly one thing, and raised the bill on every task.

How Was the Benchmark Designed?

The core idea of Reasoning Dial is to change only one variable: the same 60-problem code generation set, the same prompt, the same deterministic scorer — the only thing that changes is the value of reasoning_effort (none / low / medium / high).

The test includes three task types, deliberately designed to contradict each other:

• Deduce (logical reasoning): a 7-person, 7-day scheduling puzzle with 6-12 clues, brute-force enumerated to confirm a unique solution. This is the scenario where reasoning should help.
• Arith (arithmetic): six-step applied calculation problems with percentages and integer-division steps. This is the control group — during calibration, all models were already 100% correct at none.
• Distract (distractor counting): "you have one chisel and one power drill," followed by 1-3 irrelevant numbers, asking for the count of tools. This is the scenario where reasoning might actually hurt.

All hypotheses, prompts, and parsers were written into a pre-registration file before the official run, and the 60 test problems were locked via SHA-256 hashes. The analysis script refuses to run if the hashes don't match.

Four Knob Positions, Four Completely Different Behaviors

The first important finding: the same parameter name is not the same thing across vendors.

The median output tokens per page reveal four completely different behavior patterns:

Model	none → low → medium → high	high ÷ minimum
gpt-5.4-mini	18 → 168 → 195 → 254	×14.1
gpt-oss-120b	refused → 292 → 446 → 1,340	×4.6
claude-haiku-5.5	150 → 162 → 171 → 298	×2.0
gemini-3.7-flash	453 → 446 → 566 → 1,013	×2.2

gpt-5.4-mini's token consumption shot up from 18 to 254, a 14x amplification. Meanwhile gemini-3.7-flash already consumed a median of 453 tokens at none — more than gpt-5.4-mini's 254 at high. Gemini doesn't return reasoning traces at none and low, so it looks no different from a model that answers in one second, but you're paying for those invisible thoughts.

gpt-oss-120b flat-out refuses none, returning HTTP 400. If you write reasoning_effort=\"none\" into a multi-vendor config, one of these four models will simply error out.

Reasoning traces vs. invisible thinking: Many reasoning models generate a chunk of "thinking process" text before returning the final answer, usually surfaced as a separate field or with a special delimiter — that's the reasoning trace. It's the externalization of the model's internal chain of thought, but not every vendor exposes it to the caller. Gemini doesn't return reasoning traces at none and low; users just see a quick answer, yet billing is still based on the actual tokens generated — the hidden thinking tokens still land on your bill. To judge whether a model is "thinking invisibly," don't just look at whether the response contains reasoning text; also compare the output token count in the usage field against the length of the visible answer.

HTTP 400 and parameter value domains: Different vendors define legal values for reasoning_effort differently. The OpenAI family typically accepts none/low/medium/high, while models like gpt-oss-120b treat none as an illegal input and return HTTP 400 Bad Request directly. This means if your code hardcodes reasoning_effort=\"none\", one model in a multi-vendor routing scenario will simply fail. Production environments need to maintain a per-model whitelist of legal parameter values, or do parameter mapping and fallback at the gateway layer.

The One Real Accuracy Improvement

Across the 12 model × task combinations, only gpt-5.4-mini on the logical reasoning task showed a statistically significant accuracy improvement: from 15% to 97.5%, with a Holm-corrected p = 0.00056.

But at none, its 15% correct rate was roughly equivalent to random guessing. The main gain came from the first step: none to low added 60 percentage points, and each subsequent step added less while costing more.

The logical reasoning performance of the other two models is worth noting: Claude Haiku 5.5 already scored 97.5% at none, and Gemini was at 100%. For them, turning the knob up just means spending more on an already-solved problem.

Distractor Counting: Every Wrong Answer Overcounted

Pre-registered hypothesis H2 held that more thinking would hurt on the distractor counting task. The significance test did not support this — no model got significantly worse at high.

But the direction of the errors is interesting: every wrong distractor-counting answer, without exception, overestimated. No model ever gave a number lower than the correct answer. The models weren't failing to distinguish what tools they had — they were adding the distractor numbers in.

And the longer the response, the more likely the answer was wrong. gpt-5.4-mini's wrong answers at high had a median of 412 tokens, versus 91.5 for correct answers. gpt-oss-120b was even more extreme: wrong answers used 5,776 tokens, correct answers only 747. In the most extreme case, gpt-oss-120b spent 21,875 tokens and nearly four minutes to answer "6" when the correct answer was "5."

The researcher's framing: more effort gives the model more room to overestimate, rather than simply "making results worse."

Pre-registration and significance testing: Pre-registration means fixing and publishing (or hash-locking) the hypotheses, experimental design, prompts, scorer, and analysis script before the study begins, to prevent cherry-picking favorable conclusions after seeing the data. This article locked 60 problems with SHA-256, and the analysis script refuses to run if hashes don't match — precisely to ensure the conclusions aren't "making up a story to fit the data." Significance testing is used to judge whether observed differences could plausibly be caused by random fluctuation. The Holm correction mentioned here is a multiple-comparison correction method: when testing many model × task combinations at once, the false-positive probability accumulates, and Holm correction controls the overall error rate by progressively adjusting the p-value thresholds. So "p = 0.00056" is a result that remains significant after correction, making it more trustworthy; and "not significant" doesn't mean "no effect" — it just means the sample size wasn't enough to rule out randomness.

The Bill Never Skips a Beat

This is the data most worth showing to production decision-makers:

In 7 of the 12 combinations, high changed only the price. Accuracy was unchanged, but the cost per correct answer rose 1.5x to 3.4x.

On the control-group arithmetic task, all models' accuracy was unchanged, but the cost per correct answer rose 1.65x to 2.5x. Spending more reasoning budget on an already-solved problem buys you nothing.

gpt-oss-120b spent about 25x the cost per correct answer on distractor counting, while accuracy was 7.5 percentage points lower (not significant).

The one big winner was also expensive: gpt-5.4-mini's logical reasoning fix raised the cost per correct answer 8.5x.

Cost per correct answer: When measuring the value of reasoning effort, looking only at total spend or only at per-token price is insufficient — a more intuitive metric is "cost per correct answer," i.e., total spend divided by number of correct answers. It captures both accuracy and token consumption: if accuracy is unchanged but cost rises, the extra reasoning budget bought no accuracy gain; if accuracy rises but cost rises faster, you have to judge whether the improvement is worth it. The 1.5–3.4x, 1.65–2.5x, 8.5x, and 25x figures mentioned in this article are all multiples under this metric. On the control-group arithmetic task, all models were already at 100% accuracy, so the cost increase came entirely from superfluous thinking tokens — pure waste.

Overall Project Summary

The entire main experiment cost $4.20, with pre-experiments and calibration rounds bringing the total to about $6.38. Here are the final results for the four pre-registered hypotheses:

• H1 (the knob is not the same tool): Supported. Token scaling ranged from ×2 to ×14, gpt-oss-120b refused none, and Gemini's none ≈ low with invisible thinking.
• H2 (more thinking hurts on distractor counting): Not supported. No significant decline, but all errors were overestimates, and wrong answers were longer.
• H3 (more thinking helps deductive reasoning): Supported only for gpt-5.4-mini. 15% → 97.5%.
• H4 (when accuracy doesn't rise, the bill still does): Supported (descriptively). 7/12 combinations unchanged, costs up 1.5-3.4x.

Practical Advice

Based on a scale of 20 problems per task family, 2 repetitions, and 4 models, these are starting points, not final verdicts:

1. Test none first. Two of the four models were already near-perfect at the minimum setting.
2. If none doesn't work, try low before high. For the only model that needed the knob, none to low contributed 60 of the 82.5 points.
3. Don't assume the same setting works the same across vendors. none errors out on one model, is invisible thinking on another, and only truly turns off on a third.
4. Be wary of long answers to simple questions. On the distractor counting task, a long answer is a wrong answer.

The biggest contribution of this study isn't telling you what to set the knob to — it's revealing an overlooked fact: in most scenarios, the marginal contribution of reasoning effort to accuracy is close to zero, while its marginal contribution to cost is always positive. Before you crank it to high, see what none can give you.

One-line Summary

1,800 calls across 4 models: the reasoning effort knob fixed exactly one thing while making every task more expensive.

Paragraph Summary

A developer built the Reasoning Dial benchmark on Kaggle Benchmarks, fixing problems, prompts, and scorer while varying only reasoning_effort, completing 1,800 scored calls across 4 models. Results: of 12 model × task combinations, only gpt-5.4-mini on logical reasoning showed a significant improvement (15%→97.5%), while the rest had no statistically significant effect; 7/12 combinations showed unchanged accuracy but 1.5–3.4x higher cost per correct answer. The same parameter behaves vastly differently across vendors: gpt-5.4-mini's token consumption amplified 14x, gpt-oss-120b refused none with HTTP 400, and Gemini consumed 453 invisible reasoning tokens at none. On distractor counting, all errors were overestimates, with wrong answers consuming significantly more tokens than correct ones. Conclusion: reasoning effort's marginal contribution to accuracy is near zero, while its marginal contribution to cost is always positive.