Suppressing 'Overthinking' Tokens: How Logit Penalties Boost Qwen Model Accuracy

A Counterintuitive Finding

Large language models often fall into "overthinking" during reasoning — repeatedly saying "wait," "maybe," "perhaps," "however," constantly backtracking and second-guessing answers they've already gotten right, ultimately drifting away from the correct solution. A Reddit user, inspired by a Meta paper (arXiv:2606.00206), ran a simple but fascinating experiment: directly applying logit penalties to these "overthinking marker tokens." The result was a significant improvement in model accuracy.

The core logic is straightforward: if these hesitant, repetitive, self-doubting words tend to accompany reasoning going off the rails, then reducing their probability during decoding might make the model more "decisive" in reaching an answer.

How the Experiment Was Conducted

The experimenter randomly sampled 50 problems from the MATH-500 dataset (a math reasoning benchmark on HuggingFace) and ran them on multiple quantized versions of Qwen3.5-4B. The test models came from bartowski's GGUF quantization pack, covering five precision levels from full-precision BF16 to the heavily compressed Q2_K.

The key operation was applying a -2 bias to approximately 50 tokens via llama.cpp's --logit-bias parameter. These tokens correspond to "overthinking marker words" identified in the paper, including:

• Hesitation words: perhaps, maybe, possibly, might, could
• Pause/backtrack words: wait, Wait, hold, hmm, Hmm, reconsider, rethink, backtrack, retry, revisit
• Pivot words: however, But, although, yet, instead, Alternatively
• Self-negation words: wrong, mistake, error, incorrect, doubt, confused, uncertain, unsure

In short, the weights of words likely to trigger "let me think again" behavior are artificially lowered during generation.

Logit bias is an intervention technique applied during the decoding phase of large language models. At each generation step, the model computes a raw score (logit) for every token in the vocabulary, converts these into a probability distribution via softmax, and then samples from that distribution. The --logit-bias parameter lets users directly add or subtract a fixed value from specified tokens' logits before the softmax step. Applying -2 means the target token's raw score is reduced by approximately 2 units, dropping its probability to roughly 1/7 of the original (e^-2 ≈ 0.135) — but it doesn't completely block the token from appearing, which distinguishes it from outright banning (setting -inf). This method requires no modification to model weights, takes effect only at inference time, can be toggled on or off at any moment, and incurs virtually no additional computational overhead.

GGUF quantization format is a model storage standard introduced by the llama.cpp project that compresses model size and memory usage by reducing the numerical precision of parameters: BF16 uses 16 bits per parameter, Q8_0 approximately 8 bits, Q4_K_M approximately 4 bits, Q3_K_M approximately 3 bits, and Q2_K approximately 2 bits. Lower precision means faster inference and less memory usage, but typically at the cost of reduced model expressiveness and reasoning stability.

Results: Across-the-Board Improvement from BF16 to Q2_K

The results were quite surprising — not only did lower-precision quantized models benefit, but even the full-precision BF16 model showed a clear accuracy improvement.

Precision Format	Baseline Accuracy	With Penalty	Reasoning Token Change
BF16	74%	84%	−19.4%
Q8_0	76%	80%	−11.0%
Q4_K_M	60%	66%	−14.8%
Q3_K_M	52%	66%	−17.5%
Q2_K	12%	24%	−11.5%

Two signals stand out. First, accuracy improved across the board — Q3_K_M jumped from 52% to 66%, the largest gain, while even the previously struggling Q2_K doubled from 12% to 24%. Second, reasoning token counts dropped across all variants by 11% to 19% — the models not only got more answers right, they did so more efficiently, implying shorter reasoning chains and lower inference costs.

These two metrics improving in tandem neatly validates the "overthinking" hypothesis: the tokens the model spends on hesitation words often fail to produce correct answers and instead send the model spiraling into confusion.

Why It Works

Mechanistically, reasoning models learn during training to improve performance through long chain-of-thought sequences — but this capability is a double-edged sword. When a model develops "wait, maybe I'm wrong"-style self-doubt about an intermediate conclusion that was already correct, it may discard the right answer and venture down an incorrect branch.

Applying negative biases to these marker words effectively adds a "decisiveness" constraint to the model's reasoning process, reducing pointless backtracking and redundant verification. This corrective effect tends to be most pronounced for more heavily quantized models, which are inherently less stable in their reasoning — because lower-precision models are more prone to spinning out of control during moments of hesitation.

This touches on a deeper tension in how reasoning models are trained. Reasoning models like DeepSeek^Ad-R1 and the Qwen3 series are typically trained via reinforcement learning with verifiable rewards (RLVR) or supervised fine-tuning on chain-of-thought data, which encourages the model to generate longer "thinking processes." The training signal rewards trajectories that ultimately produce correct answers, inadvertently teaching models to use repeated verification as a safety net — even when the first derivation was already correct, the model tends to run through the verification process again. Meta's original paper (arXiv:2606.00206, titled Thinking LLMs) categorizes this behavior as overthinking, finding that when certain tokens appear — especially backtracking words — the probability of subsequent reasoning going wrong is significantly higher than baseline. From an information-theoretic perspective, these hesitation tokens trigger the model to enter a "re-evaluation" mode, which tends to do more harm than good in math reasoning tasks when applied to intermediate results that have already converged.

Reasons for Caution

The experimenter himself clearly listed caveats: this is only a single model, single run, 50-question result with a small sample size and potential for random variation. It's better characterized as a suggestive, heuristic finding worth further investigation rather than a conclusion that can be directly generalized to all models and tasks.

Several questions remain to be clarified:

• This penalty word list targets math reasoning tasks — would it be equally effective in open-ended Q&A, code generation, or other domains?
• Could over-suppressing pivot words like "however" and "but" backfire on problems that genuinely require weighing multiple perspectives?
• Different model families (e.g., non-Qwen models) have different token mappings, so penalty schemes would need to be recalibrated.

Nevertheless, the value of this experiment lies in providing a nearly zero-cost inference optimization — no retraining required, just a few extra parameters at inference time, with a chance to squeeze out additional accuracy. This is especially attractive for local deployment and resource-constrained scenarios. Anyone wanting to verify the results can reproduce this --logit-bias configuration directly in their own llama.cpp environment and see whether it helps for their particular use case.