RCD: Cracking Query Shortcuts in Chain-of-Thought with Contrastive Decoding

Chain-of-Thought (CoT) prompting teaches large models to "write out their reasoning process," but a long-overlooked problem is this: the final answer a model gives may not actually depend on the reasoning steps it generated itself. The paper Mitigating Query Shortcuts in Chain-of-Thought with Context-Aware Decoding focuses precisely on this crack—the model may bypass reasoning and answer directly based on the "query shortcuts" carried by the question itself.

The "False Dependence" Problem of Chain-of-Thought

The popularity of CoT rests on a simple assumption: generating intermediate reasoning steps improves the quality of the final answer. But recent research has repeatedly pointed out that the final answer is not always supported by the rationale (reasoning process). The model may have memorized cues related to the surface form of the question, or exploited statistical associations in the training data to take a shorter path to the answer.

The authors call this type of behavior "query-specific shortcuts." Its danger lies in its subtlety: the reasoning text looks plausible, and the answer may even be correct, but there is no causal connection between the two. Once the question is rephrased, or the reasoning steps are replaced, the answer may collapse.

So the question is not whether CoT "looks like it is reasoning," but whether the answer "behaviorally depends" on the reasoning content. This paper aims to solve exactly how to pull the answer back onto the reasoning path without retraining the model.

Moving Contrastive Decoding onto the Reasoning Chain: The RCD Method

The method proposed by the authors is called Rationale Contrastive Decoding (RCD), inspired by the existing Context-Aware Decoding (CAD). CAD was originally used to make generated outputs more dependent on a given context by contrasting the next-token distributions under "with context" and "without context" conditions, amplifying tokens whose probability rises because of the context.

RCD transplants this idea to the CoT setting: it contrasts the model's next-token prediction distributions under "with rationale" and "without rationale" conditions, boosting tokens that become more likely because of the reasoning process.

In the authors' words, RCD "amplifies tokens whose probability is increased by the rationale by contrasting the next-token distributions under with-rationale and without-rationale conditions." The entire process happens at the decoding stage and is a training-free method that requires no additional annotation or fine-tuning.

Contrastive Decoding itself is a general technique for adjusting token probabilities at the decoding stage. Its core operation is to compute the difference between two distributions: an "expert" distribution (e.g., a large model under sufficient conditions) and an "amateur" distribution (e.g., a small model or under constrained conditions), then apply temperature scaling or truncation to the difference, thereby suppressing tokens that have high probability in the amateur distribution but low probability in the expert distribution. This approach requires no additional training and can guide generation solely by modifying the logits. CAD concretizes it as a contrast between "with context vs. without context," and RCD further replaces the "condition" with the rationale, so the general framework of contrastive decoding provides a direct foundation for understanding RCD.

Theoretical Perspective: KL-Regularized Reward Maximization

The paper provides a theoretical explanation for RCD's decoding rule: it can be viewed as a reward maximization objective with a KL regularization term.

The significance of this perspective is that it elevates "making the answer more dependent on reasoning" from an engineering trick to an optimization problem backed by an objective function. The role contrastive decoding plays here is to balance the original language model distribution and the "rationale-conditioned" distribution, avoiding excessive deviation from the model's own prior.

For practitioners, this explains why RCD can change answer behavior without breaking the fluency of the language model—it does not forcefully rewrite the output, but rather weights along the reasoning signals the model has already encoded.

KL-regularized reward maximization is a classic objective form in reinforcement learning and sequence generation: maximizing expected reward while constraining the KL divergence between the current policy and a reference policy, preventing the policy from deviating too far from the prior. In language model decoding, this framework is often used to explain the practice of "adjusting token distributions under given conditions"—treating "answers depending on reasoning" as the reward signal, while the KL term ensures the generated output retains the language model's own fluency and diversity. RCD's decoding rule can be seen as an approximate solution to this objective: the tokens amplified by contrastive decoding are exactly those whose probability rises under the rationale condition, equivalent to striking a trade-off between reward (depending on reasoning) and regularization (maintaining the prior).

Experiments: Accuracy Improvements on Five Benchmarks

The authors validated RCD's effectiveness on five reasoning benchmarks. The results show that RCD improves answer accuracy compared to standard CoT.

More informative are the intervention experiments. The authors performed interventions with real rationales and random rationales respectively, observing whether the final answer's behavioral dependence on the rationale content changes. The conclusion: RCD increases the degree to which the final answer depends on the reasoning content.

This point is more critical than the accuracy numbers alone. Accuracy improvements may come from many causes, but "the answer changes as the rationale changes" directly indicates that the model is pushed toward a path that relies more on reasoning at the decoding level, while the weight of query shortcuts is relatively weakened.

In the summary, the authors write that the experiments "show that RCD improves answer accuracy and increases the behavioral dependence of final answers on rationale content," and further propose that contrastive decoding is a "simple, training-free" shortcut mitigation solution.

Behavioral dependence is an operationalized metric for evaluating reasoning faithfulness: instead of directly checking the logical consistency between the reasoning text and the answer, it intervenes in the reasoning content (e.g., replacing it with a random rationale) and observes whether the final answer changes accordingly. If the answer is sensitive to the replacement of the rationale, it indicates that the model behaviorally depends on reasoning; otherwise, it may depend on query shortcuts. This kind of intervention experiment reveals the model's internal decision path better than accuracy alone, because it directly tests the causal effect of "changing part of the input—does the output change accordingly," similar to intervention analysis in causal inference.

Implications for Deploying Reasoning Models

This work has several direct implications for the practical deployment of reasoning models.

First, the shortcut problem is an evaluation blind spot. If you only look at final answer accuracy, it is hard to judge whether the model is really using reasoning. RCD's intervention analysis provides a way to test this: change the rationale and see whether the answer follows.

Second, training-free means low cost. Compared to constructing counterfactual data for fine-tuning, the contrastive method at the decoding stage can be directly layered onto existing models and prompting pipelines, making it suitable for rapid validation.

Third, the method still has boundaries. What RCD amplifies is the reasoning signal already encoded in the model itself; if the model has no usable reasoning knowledge for a certain type of problem, contrastive decoding can hardly conjure a correct path out of thin air. The paper's experiments are concentrated on five reasoning benchmarks, and cross-domain generalization still needs more validation.

From a broader perspective, RCD partially shifts the question of "whether the model is really reasoning" from the training stage to the decoding stage. This may not be the endpoint, but it offers a pragmatic middle path: do not change the model parameters, only change the model's allocation of attention along the reasoning chain.

One-line Summary

RCD uses training-free contrastive decoding to weaken query shortcuts and increase CoT answers' dependence on reasoning.

Paragraph Summary

The paper addresses the disconnect between reasoning steps and final answers in chain-of-thought by proposing the Rationale Contrastive Decoding (RCD) method. Inspired by context-aware decoding, this method contrasts the next-token distributions under "with rationale" and "without rationale" conditions at the decoding stage, amplifying tokens whose probability rises due to the reasoning process, thereby suppressing the model's dependence on query shortcuts. The authors provide a theoretical explanation from the perspective of KL-regularized reward maximization and validate on five reasoning benchmarks that RCD improves accuracy over standard CoT. Intervention experiments further show that RCD increases the behavioral dependence of final answers on rationale content, indicating that the model is pushed toward a path that relies more on reasoning. The method is training-free and can be directly layered on, but it depends on the reasoning signals already encoded in the model itself, and cross-domain generalization still needs validation.