Hybrid Linear Attention

An attention architecture that retains standard attention in some layers while using linear attention in the remaining layers, accumulating historical information with fixed-dimensional recurrent states. It balances local precise retrieval and long-range memory, with compressed representation dimensions far smaller than the KV cache.