Reverse Self-Distillation: Weakening a Model Can Actually Improve Its Reasoning

Self-distillation offers a path to improving reasoning models without relying on external teachers, but conventional approaches typically require privileged information to construct a stronger self-teacher, which limits their applicability when only unlabeled prompts are available. A recent paper, "Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models," proposes a counterintuitive idea: instead of building a stronger teacher, deliberately weaken the model and use the contrast between the weakened and original versions as the training signal. The method, called WISD (Weakening-Induced On-Policy Self-Distillation), achieves significant gains on mathematical and code reasoning tasks.

Core Idea: From "Stronger Teacher" to "Weaker Copy"

The classic self-distillation paradigm has a stronger teacher model guide the original model, but strong teachers often depend on additional annotations or privileged information. WISD flips this: it does not pursue a stronger teacher but instead constructs a weakened version of the original model, obtaining a training signal by contrasting the behavior of the original and weakened models on the same input.

The key to this design is that the weakened model still retains part of the original model's capability, but its output quality is degraded. By comparing against the weakened version, the original model "sees" the mistakes it makes and adjusts its own reasoning strategy. The entire process requires no external teacher or privileged information and can run using only unlabeled prompts.

The paper explores two ways to construct the weakened model: low-bit quantization and magnitude pruning. Both are common model compression techniques, but here they serve as tools for generating a "weaker self" rather than for deployment optimization.

Low-bit quantization and magnitude pruning are two common model compression techniques. Low-bit quantization converts model weights from high precision (e.g., FP16) to low precision (e.g., INT8, INT4), introducing quantization error that degrades output quality while reducing storage and computation overhead. Magnitude pruning removes unimportant connections or parameters based on the absolute value of weights; the higher the pruning rate, the greater the loss of model capacity. In WISD, these two operations are not used for deployment acceleration but as "controllable weakeners": by adjusting the quantization bit-width or pruning ratio, one can precisely control the degree of capability decay in the weakened copy, thereby providing contrast signals of varying difficulty to the original model. This reverse use of compression techniques for training-signal generation is the key design that distinguishes the method from traditional self-distillation.

On-Policy self-distillation emphasizes that training signals come from trajectories the model itself generates in real time on a given input, rather than from a fixed offline dataset. In WISD, the original model and the weakened model each reason over the same unlabeled prompt, producing two parallel output trajectories. The comparison happens in policy space rather than answer space: the original model adjusts its policy by identifying deviations in the weakened copy's reasoning steps, so the training distribution stays consistent with the model's current behavior, avoiding the distribution shift common in offline distillation. This is also why WISD needs only unlabeled prompts—it exploits the model's own behavioral differences rather than externally annotated correct answers.

Experimental Setup and Main Results

WISD was evaluated across multiple architectures and scales, covering mathematical and code reasoning benchmarks. Results show that both quantization- and pruning-based weakening yield consistent average accuracy gains: 4.0 percentage points for quantization and 5.5 percentage points for pruning. Even without any privileged information, WISD reaches state-of-the-art performance.

Specific cases illustrate the point further. Pruning raises Qwen3.5-4B's accuracy on HMMT 2025 from 57.7% to 70.5%, and on LiveCodeBench v6 from 39.1% to 56.7%. Quantization raises Qwen3.6-35B-A3B's accuracy on the same benchmarks from 68.5% to 74.6% and from 61.8% to 69.5%, respectively. These gains are substantial for reasoning tasks, especially considering that the method introduces no additional supervision signal.

Interestingly, pruning delivers a higher average gain than quantization, and performs especially well on smaller models. This may be because pruning weakens model capability in a more structured way, producing clearer contrast signals.

Why Does "Weakening" Actually Work?

Intuitively, the weakened model is a version that "knows a little less." By comparing, the original model can identify which reasoning steps the weakened version deviates on, and this deviation information is more instructive than a correct answer alone. Compared with approaches that require a strong teacher, WISD's advantage is that the weakened model is always homologous with the original model, avoiding the distribution mismatch between teacher and student models.

Moreover, the weakening operation itself is stochastic and controllable. Both quantization and pruning have adjustable strength, giving the training process a flexible tuning dimension. The paper does not discuss theoretical guidance for choosing strength, but experiments show both weakening methods are effective across multiple model scales, indicating a degree of robustness.

Teacher-student distribution mismatch is an inherent challenge in traditional knowledge distillation: a strong teacher model is usually trained on larger-scale data or stronger supervision, and its output distribution systematically differs from the student model's, so a student imitating the teacher may learn reasoning patterns that do not fit its own capability boundary. WISD fundamentally eliminates this mismatch by making the weakened copy homologous with the original model—both share the same pretrained knowledge, tokenization, and reasoning style, with differences arising only from the capability decay introduced by compression. This homology focuses the contrast signal on "error patterns caused by capability weakening" rather than on architectural or data bias between teacher and student, thereby improving the purity and transferability of the training signal.

Limitations and Outlook

WISD has so far been validated only on mathematical and code reasoning tasks; whether it applies to broader natural language reasoning tasks remains unclear. The choice of weakening strength also lacks theoretical grounding and may require tuning for different models and tasks. In addition, how exactly the contrast signal between weakened and original models translates into training gradients is explained only intuitively in the paper; future work could further analyze the underlying mechanism.

Nevertheless, WISD offers a new perspective on self-distillation: you don't necessarily need a stronger teacher—a controllable weakened copy can also drive model improvement. This is especially valuable for scenarios lacking external supervision resources and opens a low-cost path for model self-improvement.

One-line Summary

WISD achieves privileged-information-free self-distillation by contrasting a weakened model copy with the original, significantly improving reasoning tasks.

Paragraph Summary

WISD (Weakening-Induced On-Policy Self-Distillation) proposes a counterintuitive self-distillation approach: instead of building a stronger teacher, it deliberately weakens the original model (via low-bit quantization or magnitude pruning) and uses the behavioral difference on the same unlabeled prompt as the training signal. The method achieves consistent gains on mathematical and code reasoning benchmarks, with quantization improving average accuracy by 4.0 percentage points and pruning by 5.5. In specific cases, pruning raises Qwen3.5-4B on HMMT 2025 from 57.7% to 70.5% and on LiveCodeBench v6 from 39.1% to 56.7%. Because the weakened copy is homologous with the original model, WISD avoids the teacher-student distribution mismatch and requires no external supervision signal. The method is currently validated only on reasoning tasks, the choice of weakening strength lacks theoretical guidance, and the mechanism explanation remains intuitive.