Language Subspace Interventions: The Truth Beyond Accuracy

Language-subspace interventions are seen as a means to improve multilingual reasoning: suppressing language-related components in the model's intermediate-layer activations. But are such methods really effective? A new paper systematically examines three key questions across the MATH and MGSM math reasoning datasets on four open-weight large language models: Do the estimated language subspaces capture translation-related variation? Can projecting out these subspaces remove language information? And can they improve accuracy without harming the response language?

Methods: Mean-SVD and DiffPCA

The study adopts two methods for constructing language subspaces: the existing Mean-SVD construction, and DiffPCA, proposed by the authors. Both aim to separate language-related directions from model activations.

On the first question, both subspaces capture a considerable degree of translation-related variation and exhibit similar patterns across different model depths. This means that regardless of which method is used, the language signal can be "caught" to some extent.

Linear probes are a commonly used interpretability analysis tool: a simple linear classifier is trained on the model's intermediate-layer activations to detect whether certain information (such as language category) remains linearly separable. If the probe can predict the target attribute from the activations with high accuracy, it indicates that the attribute information is encoded linearly in the activations. In this paper, the researchers use linear probes precisely to test whether the projection operation truly removes language information—if the probe can still perfectly classify language after projection, it means the subspace does not exhaust the language signal. The advantage of linear probes is that they are simple and interpretable, but they can only detect linearly separable information and cannot rule out the possibility of nonlinear encoding.

Mean-SVD and DiffPCA are two methods for extracting language subspaces from model activations. Mean-SVD performs singular value decomposition (SVD) on the mean activation vectors of different languages and takes the principal singular vectors as language directions; DiffPCA first computes the activation differences of the same semantics across different languages, then performs principal component analysis (PCA) on the difference matrix to capture the directions most relevant to language conversion. Both assume that language information is concentrated in a low-dimensional subspace, so it can be removed by projection. However, the choice of subspace dimensionality and whether it covers all language variation both affect the intervention effect, which is why this paper tests whether it exhausts the language information.

Language Information Can Still Be Perfectly Classified After Projection

The second question is more critical: after projecting out the subspace, does the language information really disappear? The researchers project out each subspace and then train linear probes to classify language from the projected activations.

The results are surprising: the probes can still almost perfectly identify the language. This shows that neither Mean-SVD nor DiffPCA exhaustively captures language information. In other words, the projection operation removes only part of the language signal, not all of it.

The Trade-off Between Accuracy Gains and Language Fidelity

The third question focuses on intervention effects. The researchers evaluated multiple intervention configurations on Qwen2.5-7B, varying the subspace creation method and the intervention operator.

They found that the intervention that improved MGSM accuracy the most was precisely the one that sacrificed response-language fidelity. Only one intervention slightly improved joint accuracy—that is, the answer is correct and in the target language—by only 1.60 points.

More interestingly, a minor change to the prompt alone can reproduce the accuracy gain, but language fidelity is significantly damaged. This suggests that the accuracy improvement may not come from a causal intervention on the language subspace, but from other factors.

Joint accuracy is a key evaluation metric proposed in this paper, referring to the proportion of cases where the model's answer is both correct and in the target language. Traditional evaluation focuses only on answer correctness, which may mask the phenomenon of models "cheating" to improve accuracy by switching to a language they are better at (such as English). Joint accuracy penalizes both language errors and answer errors, so it reflects multilingual reasoning ability more faithfully. This paper finds that most interventions improve MGSM accuracy but sacrifice language fidelity, leaving joint accuracy almost unchanged or even lower; only one intervention improves joint accuracy by 1.60 points, indicating a clear trade-off between accuracy and language fidelity.

Intervention Evaluation Needs to Go Beyond Accuracy

The authors' core argument is that causal interventions must be judged on more metrics than accuracy alone, so as to fully account for the trade-off of language substitution.

In multilingual reasoning, models may improve accuracy by switching to a language they are better at (such as English), but this defeats the purpose of multilingual applications. Therefore, when evaluating language subspace interventions, both accuracy and language fidelity need to be examined to avoid "spurious improvements."

This study reminds us that while pursuing higher accuracy, we must not ignore whether model behavior meets expectations. Intervention methods for multilingual reasoning require a more comprehensive evaluation framework.

One-line Summary

Language subspace interventions do not exhaust language information, and accuracy gains may come at the cost of language fidelity.

Paragraph Summary

This paper systematically examines the effectiveness of language subspace interventions in multilingual reasoning. The study uses Mean-SVD and the authors' proposed DiffPCA to construct language subspaces, testing three questions on four open-source large models and the MATH and MGSM datasets: whether the subspace captures translation-related variation, whether language information is removed after projection, and whether the intervention can improve accuracy. The results show that although both subspaces capture some translation variation, linear probes can still almost perfectly classify language after projection, indicating that language information is not exhaustively removed. On Qwen2.5-7B, the intervention that improves MGSM accuracy the most sacrifices response-language fidelity, and only one intervention improves joint accuracy by 1.60 points; a minor prompt modification alone can reproduce the accuracy gain, but language fidelity is significantly damaged. The authors argue that intervention evaluation must consider both accuracy and language fidelity to avoid spurious improvements caused by language substitution.