Competing Learning in Hybrid Architectures: Unraveling the Tug-of-War Between Recurrence and Attention

Introduction: The Rise of Hybrid Architectures and Hidden Concerns

In recent years, the design of large language models has gradually shifted from a single Transformer architecture to hybrid architectures—combining linear-time recurrence with softmax attention. Recurrent layers compress context into a fixed-size state, providing a highly expressive mixing mechanism with guaranteed high throughput; softmax attention, meanwhile, performs better on contextual retrieval tasks. However, this hybrid design harbors a key contradiction: different types of mixers compete to learn the same function, and once one layer learns a function, it suppresses the learning signal for the corresponding function in other layers. This paper reveals this "Competing Learning" phenomenon and proposes a staged training strategy to mitigate the issue.

The difference between linear-time recurrence and softmax attention: Linear-time recurrent layers (such as the recurrence mechanisms in Mamba and RWKV) run inference with O(1) per-token state update complexity, compressing historical context into a fixed-dimensional hidden state, and thus have constant memory footprint and high throughput; however, their ability to perform precise long-range retrieval is limited by state capacity. Softmax attention retains the key-value pairs of all historical tokens and aggregates them via query-key similarity weighting, enabling precise retrieval of information at any position, but its computation and memory grow quadratically with context length. Hybrid architectures stack the two alternately, attempting to combine the efficiency of recurrent layers with the precise retrieval capability of attention layers. Understanding this trade-off is the prerequisite for grasping the "competing learning" problem in this paper: the two types of layers overlap functionally, and whoever learns retrieval first may suppress the other's learning signal.

Competing Learning: How Recurrent Layers "Jump the Gun" on Attention

Through layer attribution analysis, the paper finds that under typical data distributions, modern recurrent layers learn the retrieval function faster than softmax attention. This means that in early training, recurrent layers quickly master retrieval capability and thereby suppress the signal for attention layers to learn the same function. As a result, attention layers in most cases fail to effectively learn retrieval, impairing the overall performance of the hybrid architecture.

This competitive phenomenon is not accidental but stems from differences in the optimization dynamics of the two types of mixers. Because recurrent layers have more direct parameter update paths, they can capture retrieval patterns in the data more quickly; attention layers, by contrast, require more complex interactions to learn the same function. Once recurrent layers "jump the gun" successfully, attention layers lose their learning incentive, creating a "winner-takes-all" situation.

Layer attribution analysis: This is a technique for quantifying each layer's contribution to the final output in a network. Common approaches include gradient-based attribution (such as integrated gradients), activation patching, or ablation experiments—i.e., masking or replacing a layer's output layer by layer and observing changes in model performance on a specific task (such as contextual retrieval). This paper uses the method to locate the following: during training, the attribution score of recurrent layers on the retrieval task rises rapidly and becomes dominant, while the attribution score of attention layers remains low for a long time. The value of attribution analysis lies in partially extracting "which layer is learning what" from the black box, turning "competing learning" from speculation into an observable phenomenon, and providing an intervention basis for subsequent staged training.

Staged Training: Giving Attention a Chance to Learn Too

Based on the above findings, the paper proposes a simple yet effective solution: during training, stop updating the weights of recurrent layers in the middle phase. In this way, recurrent layers no longer "monopolize" the learning signal, and attention layers can gradually learn the retrieval function in subsequent training. Specifically, training is divided into two stages: in the first stage, all layers are trained normally so that recurrent layers quickly master retrieval; in the second stage, the weights of recurrent layers are frozen and only attention layers and other parts are updated, thereby creating learning space for attention layers.

This staged training strategy was validated on natural language pretraining tasks. The experiments used models with up to 1 billion parameters, and the results show that the method brings significant improvements in language modeling loss, commonsense reasoning, and various retrieval-related tasks, and is effective across different context lengths.

Weight freezing and training scheduling: Freezing weights means setting requires_grad to False for some parameters during a training stage so that they do not receive gradient updates. The strategy in this paper is not permanent freezing but temporary freezing of recurrent layers in the middle of training, allowing attention layers to obtain exclusive learning signals in the remaining training. This differs from the common "train the backbone first, then fine-tune the head" or "progressive unfreezing": the latter are usually motivated by stability or transfer learning, whereas the purpose of freezing here is to reallocate learning opportunities and resolve inter-layer competition for the same function. This operation does not change the model structure or affect forward computation at inference, so deployment cost is zero; the only things that need tuning are the start timing and duration of freezing, and the paper uses ablation experiments to show that mid-training intervention is more effective than freezing throughout or freezing late.

Experimental Validation: Significant Improvements Across Multiple Tasks

The paper's empirical results cover multiple dimensions. On language modeling loss, staged training achieves lower loss values than standard hybrid architecture training; on commonsense reasoning tasks (such as HellaSwag and PIQA), the model also performs better; and on retrieval-related tasks (such as question answering and fact checking), the improvements are particularly pronounced. These gains remain consistent across different context lengths, indicating that the method has good generalization ability.

As a detail, staged training does not increase the number of model parameters or inference cost; it only introduces a simple pause in weight updates during the training phase, and therefore has high practical value.

Implications and Outlook: New Ideas for Hybrid Architecture Design

This study reveals the competitive relationship among different mixers in hybrid architectures and provides a new perspective for understanding model training dynamics. Traditional training methods assume that all layers learn collaboratively, but the competing learning phenomenon shows that there may be a "zero-sum game" between layers. Staged training, as a lightweight intervention, can effectively mitigate this problem and improve the final performance of hybrid architectures.

In the future, this idea may be extended to more complex hybrid architecture designs, such as dynamically adjusting the update pace of different layers or introducing more fine-grained attribution analysis to guide training strategies. For AI practitioners, this means that when adopting hybrid architectures, they need to pay attention to the potential risks brought by inter-layer competition and consider using training scheduling to unlock the potential of each component.

One-line Summary

Reveals the competing learning phenomenon between recurrent layers and attention layers in hybrid architectures, and proposes a staged training strategy to mitigate it.

Paragraph Summary

This paper studies a neglected training dynamics problem in hybrid architectures composed of linear recurrence and softmax attention: the two types of mixers compete to learn the same retrieval function. Layer attribution analysis shows that recurrent layers learn retrieval faster than attention layers and successfully "jump the gun" in early training, thereby suppressing the signal for attention layers to learn retrieval and impairing the overall performance of the hybrid architecture. The authors propose a staged training strategy—freezing the weights of recurrent layers in the middle of training and updating only attention layers and other parts—to create learning space for attention layers. On models with up to 1 billion parameters, the method achieves significant improvements in language modeling loss, commonsense reasoning, and retrieval-related tasks without increasing parameter count or inference cost, and has high practical value.