Where Do Distilled Students Hide Their 'Thinking'? Uncovering the Weight Subspace

Large Reasoning Models (LRMs) excel at solving complex problems with long chain-of-thought reasoning, but when distilling these capabilities into smaller models, the student models often struggle on long solutions. Recent research shortens teacher outputs by interpolating between an LRM and a short-answer model at a ratio λ, thereby training the student. However, a fundamental question remains unanswered: when the teacher no longer 'thinks,' what exactly is preserved in the student's weights?

This article is based on a cutting-edge study that analyzes the weight subspace associated with 'thinking' behavior in distilled student models. The research compares LoRA adapters from three teacher-student pairs, analyzing changes in the update subspace across multiple λ values, and for the first time locates the weight region that stores 'thinking' behavior while revealing its causal connection to model output behavior.

Research Background: The Thinking-Length Dilemma in Distillation

Large reasoning models improve answer accuracy by generating detailed intermediate reasoning steps (i.e., 'thinking'). But when distilling this capability into smaller models, students struggle to reproduce long solutions, leading to performance degradation on tasks requiring long-chain reasoning. To mitigate this, a common approach is interpolation: mixing an LRM with a short-answer model at a ratio λ to generate teacher outputs of varying lengths for training the student.

Although interpolation effectively shortens solution length, what the student inherits from the teacher is not just answers but also patterns of 'how to think.' When the teacher stops generating explicit thinking steps, which parts of the student's weights still retain traces of thinking? This is key to understanding the distillation mechanism.

Method: Comparing LoRA Adapters Across Teacher Pairs

The research team trained three student models, each corresponding to a different teacher pair (each pair consisting of an LRM and a short-answer model). Each student was fine-tuned using LoRA adapters, and adapter weights were saved at multiple interpolation ratios λ. By comparing changes in the adapter update subspace across different λ values, the researchers sought to locate the weight directions associated with thinking behavior.

Specifically, they computed the principal components of the adapter update matrices and tracked how these directions evolved with λ. When λ adjustment caused the teacher to stop thinking and solution length to shorten, certain directions in the update subspace shifted significantly. This shift point coincided exactly with the disappearance of the teacher's thinking behavior, thereby locating the 'place' where the change occurs without direct attribution.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that freezes pre-trained model weights and injects trainable low-rank matrices (typically the product of two small matrices) into each layer, adapting to downstream tasks with very few parameters. The LoRA adapter's update matrix can be viewed as a low-dimensional representation of model behavior changes, and its principal component analysis can reveal the main directions adjusted during fine-tuning. In this paper, the researchers locate the weight directions associated with 'thinking' behavior by comparing changes in the LoRA update subspace across different interpolation ratios λ—precisely leveraging LoRA's compactness and interpretability.

Core Finding: A Shared 'Thinking' Subspace

The analysis revealed a shared adapter subspace closely associated with three types of thinking-like behavior: reflective tokens (e.g., 'wait,' 'let me check'), monologue format (step-by-step narration), and long solutions. This subspace was prevalent in adapters before the teacher stopped thinking and was shared across multiple teacher pairs.

Interestingly, when a teacher pair halved its solution length but still retained thinking tokens (such as <think> tags), the subspace retained about half (based on post-hoc analysis). However, once the teacher completely stopped thinking, the subspace was almost entirely emptied. This indicates that the subspace's persistence is directly related to whether the teacher generates thinking tokens, rather than solely depending on solution length.

Further experiments showed that removing this subspace significantly shut down thinking-like behavior across all teacher pairs, students, and benchmarks. But on one teacher pair and another student, removing this subspace left significantly fewer reflective tokens than removing the largest principal component in the update matrix; in other cases, random directions of equal energy often achieved similar effects. This means that although the subspace is critical, it is not the sole determinant—its role may be redundant with other directions.

Principal Component Analysis (PCA) is a commonly used dimensionality reduction and feature extraction method that projects data onto directions of maximum variance (principal components) through orthogonal transformation. In this paper, the researchers perform PCA on the LoRA adapter update matrices, extract their main directions of variation, and track how these directions change with the interpolation ratio λ. The energy (variance contribution) of a principal component reflects its importance in the overall update. Removing the largest principal component or a specific subspace allows observation of its impact on model behavior, thereby inferring the functional relevance of that subspace.

Causal Validation: Transplant and Removal Experiments

To verify the causal relationship between the subspace and thinking behavior, the researchers conducted two sets of intervention experiments.

Removal experiment: After removing the shared subspace from student adapters, the frequency of the model generating reflective tokens and long solutions dropped dramatically, and thinking-like behavior almost disappeared. This effect was consistent across three students and multiple benchmarks, indicating that the subspace is necessary for maintaining thinking behavior.

Transplant experiment: When the subspace was grafted onto a short-answer student model with double weight, the model began generating early reflective tokens (based on post-hoc counts), similar to the effect of the full adapter. But interestingly, the student model rarely completed full solutions, often interrupting after generating tokens. This suggests that the subspace can trigger the 'initiation' of thinking but is insufficient to sustain a complete reasoning chain.

Together, these experiments show that the subspace is not only correlated with thinking behavior but also has a causal effect on it. However, its effect is influenced by the overall state of the model, and transplanting it alone cannot fully reproduce the complete behavior.

Causal intervention in neural network interpretability research typically refers to artificially modifying the model's internal state (such as removing, amplifying, or transplanting specific weight directions) to observe behavioral changes, thereby establishing a causal relationship between internal representations and behavior. The removal and transplant experiments in this paper belong to this category: removing the shared subspace to test its necessity, and transplanting the subspace to test its sufficiency. This intervention approach reveals the functional role of weight subspaces more effectively than purely observational analysis, but it should be noted that intervention effects may be influenced by the overall state of the model.

Discussion and Implications: What This Means for Distillation Practice

This study is the first to locate a subspace in weight space associated with distilled thinking behavior, providing a new perspective on how small models inherit the reasoning capabilities of large models. Its findings have multiple implications for model distillation practice.

First, the choice of interpolation ratio λ requires caution: when λ causes the teacher to stop thinking, the thinking-related subspace in the student adapter is emptied, potentially causing the student to lose some reasoning capability. If one wishes the student to retain thinking behavior, it is important to ensure that teacher outputs contain sufficient thinking tokens, not just shortened solution length.

Second, the shared nature of the subspace means that different teacher pairs may encode thinking behavior in similar ways, offering a potential pathway for cross-model knowledge transfer. But the transplant experiment also shows that the subspace alone is insufficient to sustain complete reasoning—other parts of the student model (such as foundational capabilities) are equally important.

Finally, the effect of removing the subspace is not entirely equivalent to removing all important directions, suggesting that thinking behavior may be jointly encoded by multiple redundant directions. Future research could further explore the interactions among these directions and how to distill reasoning capabilities more efficiently.

Conclusion

By systematically comparing LoRA adapters, this study identifies a shared weight subspace associated with distilled thinking behavior. This subspace is emptied when the teacher stops thinking; removing it shuts down thinking behavior, while transplanting it can trigger but not complete reasoning. These findings deepen our understanding of the distillation mechanism and provide an empirical basis for improving the reasoning capabilities of small models.

For AI researchers and engineers, this work reminds us that distillation is not just the transfer of answers but also the encoding of behavioral patterns. Understanding how these patterns are stored in weights is a key step toward designing more efficient distillation methods.

One-line Summary

Research locates a shared LoRA subspace in distilled student models that stores 'thinking' behavior and validates its causal role.