Task Vectors Can Be Linearly Decomposed into Interpretable Components for Precise LLM Control

Among steering techniques for large language models (LLMs), Task Vectors (TVs) have drawn attention for their ability to compactly guide model behavior, but they have long been treated as atomic directions derived from hidden states, with little known about their internal functional structure. A recent study proposes that task vectors are not indivisible wholes but can be linearly decomposed into functionally interpretable components, opening a new path toward more precise model control.

The Internal Structure of Task Vectors: From Atomic to Decomposable

Task vectors are typically extracted by contrasting model hidden states across different tasks, making them a lightweight means of model steering. However, the traditional view treats them as a single direction, overlooking the rich functional information they may contain. Inspired by the decomposition of in-context learning into Task Recognition (TR) and Task Learning (TL), this study attempts to break a single task vector into functionally interpretable components.

Based on the unembedding vectors of candidate labels, the researchers constructed a TR/TL decomposition framework, separating each task vector into a TR component, a TL component, and a residual component. The TR component is responsible for identifying the task type, while the TL component is responsible for learning the specific mapping relationships of the task. This decomposition not only improves interpretability but also lays the groundwork for subsequent intervention optimization.

Task Recognition and Task Learning in In-Context Learning

In-context Learning (ICL) refers to a model's ability to complete a new task using only a few examples or instructions in the prompt, without updating its parameters. Prior research has broken down the mechanism of ICL into two stages: Task Recognition (TR), which determines which type of task is to be performed based on examples or instructions, and Task Learning (TL), which induces the specific input-to-output mapping from the examples. TR relies more on semantic cues about the task type, while TL relies on patterns across examples. This study borrows this dichotomous framework to break a single task vector into functional components corresponding to TR and TL respectively, thereby giving the black-box process of "task vectors guiding model behavior" interpretable intermediate variables.

Unembedding Vectors and Candidate Labels

In Transformer language models, the unembedding (also called output embedding or LM head) matrix maps the final-layer hidden states to a logit distribution over the vocabulary. For classification tasks, the unembedding vector corresponding to a candidate label can be viewed as the "readout direction" of that label in representation space. Using the unembedding vectors of these candidate labels as a basis, the researchers project and decompose the task vector, obtaining TR/TL coefficients aligned with the semantics of specific labels. This approach connects abstract hidden-state interventions with interpretable label readouts, allowing the decomposed components to map directly onto the two functions of "identifying task type" and "learning label mappings."

Bayesian Optimization Improves Intervention Effectiveness

To make more effective use of the decomposed components, the research team adopted Bayesian optimization to search over the TR and TL coefficients in order to find better intervention directions. They evaluated this across multiple model families, classification tasks, and both instruction-based and demonstration-based task vectors.

Experimental results show that the intervention directions identified by TR/TL-based optimization outperform the original task vectors. The improvement is especially reliable for instruction-based task vectors. Although optimizing random directions can also improve performance, the TR/TL-guided search exhibits greater stability.

Bayesian Optimization and Steering Coefficient Search

Bayesian optimization is a global optimization method for expensive black-box functions. It fits a surrogate model such as a Gaussian process to the performance of evaluated points and uses an acquisition function to trade off "exploration" and "exploitation," thereby approaching the optimum with fewer evaluations. In this study, the objective to be optimized is the performance on a validation set of an intervention direction (defined by a combination of TR and TL coefficients), and each evaluation requires running model inference, which is costly—making Bayesian optimization a suitable choice. Compared with grid search or random search, it can more stably find better intervention directions in a high-dimensional coefficient space, which is one reason why TR/TL-guided search is more reliable than optimizing random directions.

Empirical Evidence of Functional Roles: Distinct Effects at the Logit Level

Beyond performance gains, the study also found that isolated interventions on the decomposed components produce different effects at the logit level, and these effects align with the components' expected functional roles. For example, intervening solely on the TR component mainly affects task-type recognition, whereas intervening on the TL component affects the specific task-learning process. This consistency further validates the reasonableness of the decomposition, indicating that task vectors do indeed contain functionally separable components.

Logit-Level Interventions and Validation of Functional Roles

Logits are the unnormalized scores the model produces over the output vocabulary, which become predicted probabilities after softmax. After applying a steering intervention to hidden states, observing changes in the logits of each candidate label can reveal which output dimensions the intervention affected. If intervening solely on the TR component mainly changes the logits of labels related to the task type, while intervening on the TL component mainly changes the logits of specific category mappings, this indicates that the decomposed components indeed serve different functions. This "intervention–logit response" correspondence provides direct empirical evidence that task vectors have a functionally separable structure, rather than merely indirect support at the performance level.

Multi-Source Validation and Consensus

According to the research team's experiments across multiple model families, as well as discussions by independent analysts (such as Bilibili creators and YouTube channels), the decomposability of task vectors has been cross-validated. The consensus is that task vectors are not indivisible wholes but are composed of functionally interpretable components, which offers new ideas for model interpretability and control. At the same time, regarding the effectiveness of the optimization method, different sources agree that TR/TL decomposition can deliver more reliable performance improvements.

Implications for AI Applications and Research

This research has important implications for both AI applications and frontier research. At the application level, more precise interventions mean that LLM behavior can be controlled at a finer granularity—for example, dynamically adjusting task recognition and learning strategies in classification tasks. At the research level, the decomposition of task vectors provides a new tool for understanding the mechanisms of in-context learning and is expected to advance the development of interpretable AI.

As large models grow increasingly complex, a structured understanding of internal representations becomes crucial. The linear decomposition of task vectors not only reveals their internal functional structure but also provides an actionable means of control, which may become one of the standard methods for future model editing and adaptation.

One-line Summary

Task vectors can be linearly decomposed into task recognition and task learning components, enabling more precise model control.

Paragraph Summary

This article introduces a recent study on Task Vectors in large language models. While the traditional view treats task vectors as atomic single directions, this study—inspired by the decomposition of in-context learning into Task Recognition (TR) and Task Learning (TL)—uses the unembedding vectors of candidate labels to break task vectors into a TR component, a TL component, and a residual component. The study further employs Bayesian optimization to search over TR/TL coefficients, finding that the optimized intervention directions outperform the original task vectors, and are especially reliable for instruction-based task vectors. Isolated intervention experiments also show that different components produce effects at the logit level consistent with their expected functions, validating the reasonableness of the decomposition. This work offers new ideas for the interpretability and fine-grained control of LLMs.