AsymLoop

The paper proposes the asymmetric recurrent Transformer (AsymLoop), which separates single-pass context encoding from recurrent state refinement.

AsymLoop uses a single-pass causal encoder to generate layer-specific global KV caches, which remain unchanged throughout the iterative process.

AsymLoop's decoder is a weight-tied recurrent module whose hidden state is updated at each iteration, and local KV is generated accordingly.

AsymLoop performs joint attention over global KV and local KV through a single softmax, so that each update integrates the persistent context with the current intermediate state.

AsymLoop's separated design brings computational efficiency: the global KV needs to be computed only once, and only the local KV and state need updating during recurrent iterations.

AsymLoop's fixed global cache provides an unchanging context anchor for the iterative process, avoiding the problem in full-stack recurrence where the context representation drifts with iteration.

The authors apply a heavy-tail-guided layer-wise learning rate method to the encoder and decoder to coordinate the different learning dynamics of the two parts.

Under matched effective depth and training token budget, AsymLoop exhibits lower validation perplexity and higher average downstream accuracy than Full Loop.

At the 1B scale, AsymLoop's average zero-shot accuracy on seven common-sense reasoning benchmarks improves from 49.58% to 50.85% (compared with Full Loop).

The authors extend the experiments to longer training budgets, OLMo-style backbones, and sparse mixture-of-experts (MoE) models, observing consistent gains in all cases.

AsymLoop proposes a design principle: different functional modules in recurrent computation should not be treated equally—context encoding is single-pass and static, while state refinement is iterative and dynamic.

Extending AsymLoop to MoE backbones means the layers in the recurrent decoder are themselves sparsely activated, further amplifying parameter efficiency.