How Fine-Tuning Reshapes LLM Internals: Decoupling Representational Change from Causal Importance

We Don't Really Know What Fine-Tuning Changes

Fine-tuning has become the go-to method for adapting large language models (LLMs) to downstream tasks. Whether the task involves classification, generation, or reasoning, developers can typically achieve significant performance gains with just a small number of parameter updates on top of a pretrained model. Yet a long-overlooked question remains: how does fine-tuning actually reshape a model's internal mechanisms? We observe performance improvements, but understand surprisingly little about the representational changes happening underneath.

This is precisely the gap addressed by a recent arXiv paper (arXiv:2609.21113). The authors systematically examine how fine-tuning affects LLM internal representations — including attention patterns and layer-wise activations — and then ask a deeper question: do these changes actually correspond to the components that causally drive task performance?

Representational Change ≠ Causal Importance

The study's core methodology relies on EAP (Edge Attribution Patching) to identify task-relevant components — specifically, particular attention heads and logit-level activations. This causal attribution approach pinpoints which internal components genuinely determine task performance, rather than simply highlighting parts that appear to change the most.

One key finding is that the critical components identified by EAP are concentrated in a specific subset of layers, suggesting a degree of "functional localisation" in how the model internalises task-specific behaviors. In other words, task capabilities are not uniformly distributed across the entire network — they tend to cluster.

The more intriguing result is the decoupling between the two: the layer-wise distribution of causally important components and the layers that undergo the most dramatic representational change during fine-tuning are largely uncorrelated. Put simply, the layers that "move the most" during fine-tuning are not necessarily the ones that causally matter for task performance. Surface-level representational drift does not align with functional importance.

This finding has direct implications for interpretability research: simply observing the magnitude of changes in activations or attention weights can be misleading when trying to understand how a model works. To truly understand what fine-tuning changes, causal analysis tools are essential.

EAP (Edge Attribution Patching) is a causal attribution technique that builds on the idea of activation patching. The basic approach is: while the model processes a given input, replace the activation values on specific edges (i.e., information pathways between components) with corresponding values from a different input (typically a "clean" vs. "corrupted" contrast pair), then measure how much the output changes. A larger change indicates a greater causal contribution from that edge to the task output. Unlike directly observing the absolute magnitude of attention weights or activations, EAP asks "what happens to performance if we remove this pathway?" — making it better at distinguishing components that are truly functional from those that are merely correlated with the task but not necessary. This approach is a core tool in mechanistic interpretability, and has been widely used in recent years to locate specific functional modules such as "induction heads" and "fact retrieval" circuits.

Component Overlap Does Not Equal Capability Transfer

The study also examined cross-task relationships and arrived at a counterintuitive conclusion. Intuitively, we might expect that if two tasks share many of the same critical components, fine-tuning on one should benefit the other — a form of positive transfer.

But the experiments show that when two tasks are fundamentally different in nature (e.g., a classification task vs. a generation task), overlap in EAP-identified components does not lead to cross-task performance transfer.

More concerning is the potential for negative effects: when two tasks share a high degree of overlap in their critical components, fine-tuning on one task can actually degrade performance on the other. This means shared components become contested resources — the reshaping of certain attention heads or activations for one task can disrupt another task's reliance on those same components.

"Component overlap" here refers to the proportion of shared entries when the high-importance attention heads and activation nodes identified by EAP for two different tasks are compared. Intuitively, sharing critical components suggests that two tasks rely on similar internal computation pathways — so fine-tuning adjustments to those shared components for one task should also help the other. However, this reasoning overlooks a key detail: the same attention head can serve entirely different sub-functions across tasks. A classification task might require it to focus on the sentiment polarity of a sentence, while a generation task might need it to capture semantic coherence. Fine-tuning that head for the former optimizes it in a task-specific direction that does not automatically benefit the latter — and may in fact damage the original encoding. This explains why "component overlap" can actually be a risk signal for negative transfer rather than a guarantee of positive transfer, especially when the tasks differ substantially in nature.

What This Means for Practitioners

While this research operates at the level of fundamental mechanisms, it carries several practical implications for model development.

First, multi-task fine-tuning warrants more caution. If target tasks share heavily overlapping causal components but differ substantially in nature, simultaneous fine-tuning may cause mutual interference rather than synergistic gains. Assessing component overlap across tasks could help anticipate potential performance conflicts.

Second, interpretability analysis must distinguish between "change" and "importance". Observing which layers change the most after fine-tuning does not directly tell you which layers matter most for the task. Causal attribution methods like EAP offer a more reliable lens.

Third, the existence of functional localisation provides theoretical grounding for parameter-efficient fine-tuning, modular editing, and related directions. If task capabilities are concentrated in specific components within specific layers, targeted interventions on those components may be more efficient than full-parameter fine-tuning.

Conclusion

This work decomposes the vague question of "what does fine-tuning change?" into two independently measurable dimensions: the degree of internal representational change, and the causal importance of components. It reveals that these two dimensions are not the same thing, and reminds us to watch for the negative interference that component overlap can introduce in cross-task fine-tuning. For researchers and engineers who want to deeply understand LLM internals — or who are doing multi-task adaptation in practice — these findings deserve serious consideration.