Reasoning Dial
A benchmark specifically designed to measure the effect of the reasoning_effort parameter. By fixing 60 code generation problems, the same prompt, and a deterministic scorer, it changes only the reasoning strength level to evaluate differences in model behavior.
Timeline (last 90 days)
The Reasoning Dial benchmark uses a set of 60 code generation problems, the same prompt, and the same deterministic scorer, with the only change being the reasoning_effort value (none/low/medium/high), and includes three task types: Deduce (logical reasoning), Arith (arithmetic), and Distract (distractor counting).
The entire main experiment cost $4.20, and including the pilot experiments and calibration rounds, about $6.38.
Final results of the four pre-registered hypotheses: H1 (the knob is not the same tool) supported; H2 (more thinking is harmful on distractor counting) not supported; H3 (more thinking helps deductive reasoning) supported only for gpt-5.4-mini; H4 (the bill still rises when accuracy does not) supported (descriptive).
All Facts (3)
The Reasoning Dial benchmark uses a set of 60 code generation problems, the same prompt, and the same deterministic scorer, with the only change being the reasoning_effort value (none/low/medium/high), and includes three task types: Deduce (logical reasoning), Arith (arithmetic), and Distract (distractor counting).
50%UnverifiedThe entire main experiment cost $4.20, and including the pilot experiments and calibration rounds, about $6.38.
50%UnverifiedFinal results of the four pre-registered hypotheses: H1 (the knob is not the same tool) supported; H2 (more thinking is harmful on distractor counting) not supported; H3 (more thinking helps deductive reasoning) supported only for gpt-5.4-mini; H4 (the bill still rises when accuracy does not) supported (descriptive).
50%