[KongchangAI]
ConceptHybrid Language Model

Hybrid Language Model

A language model that alternates or combines full attention layers with efficient layers such as linear attention and state space models within its architecture, aiming to balance performance and efficiency.

Timeline (last 90 days)

Oct 28

Hybrid language models alternate or combine full-attention layers with efficient layers such as linear attention and state space models (e.g., Mamba); full-attention layers must store key-value vectors for every historical token, with overhead growing linearly with sequence length, while linear attention layers replace the KV cache with a fixed-size recurrent state.

Unverified50%

All Facts (1)

Source Articles