Google's New Linear Attention Model Muir: 25MB State Holds 10 Million Tokens

The "State Bottleneck" of Linear Attention Revisited

While Transformer's attention mechanism is powerful, its memory footprint growing quadratically with sequence length has always been a pain point in long-context scenarios. Linear Attention attempts to compress historical information into a fixed-size state, thereby reducing complexity to linear. However, a fixed state means lossy compression of information, and how to efficiently "remember" key information in long sequences has always been the core challenge of this direction.

Google Research's recently proposed Muir model offers a new solution against this backdrop. According to a discussion thread on the Reddit community r/MachineLearning, Muir achieves storing a 10 million token context with a state of about 25MB through Sensitivity-Allocated Gating and Matched-Filter Readout. This figure comes from the original paper's abstract and has sparked discussion in the community about whether linear attention is reaching a practical inflection point.

Linear Attention and KV Cache: Standard Transformer self-attention requires saving Key and Value vectors for each token, forming a KV Cache that grows linearly with sequence length, and the attention computation itself is quadratic in sequence length. Linear attention, through kernel function approximation or recurrent forms, compresses historical information into a fixed-size recurrent state matrix, so that each step only requires constant-level computation and storage. The cost is limited state capacity, where early information in long sequences is continuously overwritten. State space models like Mamba and RWKV follow the same "fixed state" route, differing in how state transitions are parameterized and whether parallel training is supported. Understanding this makes it clear why "what exactly is retained in the state" becomes the core question of this direction.

Two Core Mechanisms: Gating Allocation and Matched Filtering

Sensitivity-Allocated Gating: Letting the State "Know" What to Remember

Traditional linear attention typically uses fixed decay or update rules to maintain the state matrix. Muir's improvement lies in introducing the concept of "sensitivity"—dynamically allocating gating resources based on how much the input content affects the final output. In other words, the model evaluates the importance of each token's information for subsequent predictions and prioritizes allocating limited state capacity to those "sensitive" pieces of information.

This design philosophy is similar to selective attention in human memory: not all inputs are equally important, and key information needs to be strengthened and retained. From an engineering perspective, sensitivity allocation means gating is no longer a static hyperparameter but a dynamic process coupled with input content, which improves the efficiency of state representation.

Matched-Filter Readout: Precise Extraction from Compressed State

If gating determines "what to write," then the readout mechanism determines "how to read." Muir adopts a Matched-Filter-style readout strategy. Matched filtering is a classic concept in signal processing, referring to using a filter matched to the signal characteristics to maximize the signal-to-noise ratio. In the attention context, this means the interaction between the query vector and the state is not a simple dot product, but a readout operation aligned with historical information patterns, thereby more precisely reconstructing the compressed context.

These two mechanisms complement each other: gating is responsible for efficient writing, and matched filtering is responsible for precise reading, jointly supporting information fidelity under ultra-long contexts.

Matched Filter: Originating from radar and communication signal processing, it refers to designing a filter that maximizes the output signal-to-noise ratio given a known signal waveform, with the optimal solution being the conjugate correlation of the signal template. Intuitively, it uses "the template most similar to the target signal" to perform correlation operations with the received signal, thereby highlighting the target amid noise. Borrowing this concept in the attention context means that the readout stage does not let the query vector perform indiscriminate weighting on the state, but rather lets the query construct a template aligned with historical information patterns and then correlate it with the compressed state, thereby more targetedly reconstructing the compressed content. The value of this analogy lies in transforming "readout" from an empirical dot-product design into a signal extraction problem backed by signal-to-noise ratio theory.

25MB State and 10 Million Tokens: What It Means

What scale is 10 million tokens? Roughly estimated in English text, this is equivalent to the volume of dozens of full-length novels; in code scenarios, it could cover the complete repository of a medium-sized project. Yet the state carrying this information is only about 25MB, even smaller than the size of many single images.

The significance of this compression ratio lies in deployment feasibility. When traditional full attention processes million-level tokens, the GPU memory overhead of the KV Cache quickly becomes a bottleneck, whereas a fixed-size state decouples memory usage from sequence length. For scenarios that need to run long-context applications on edge devices or GPUs with limited memory, this characteristic has practical appeal.

That said, a fixed state inevitably involves information loss. Muir's value lies in minimizing this loss through smarter gating and readout design. Some views in the community discussion also remind that the trade-off between compression ratio and task accuracy still needs to be validated on more downstream benchmarks.

Compression Ratio and Information-Theoretic Perspective: If 25MB is stored in bf16, that is approximately 12.5 million parameters, meaning each token can only be allocated an information budget of about 1.25 numeric bits on average. This shows from an information-theoretic standpoint that a fixed state is inevitably lossy compression, and any claim of "losslessly remembering the entire context" does not hold. Therefore, the key metric for evaluating such models is not the state size itself, but the accuracy retention rate on tasks such as retrieval, copying, and long-range reasoning under a given state budget. This also explains why the community emphasizes the need for validation on more downstream benchmarks: when the compression ratio is fixed, the accuracy differences brought by different gating and readout designs are what truly distinguish model quality.

Community Reaction and Open Questions

In the Reddit discussion, attention focused on several aspects. The first is comparison with existing linear attention variants (such as state space model routes like Mamba and RWKV)—whether Muir's sensitivity gating brings consistent accuracy improvements, or is only effective on specific retrieval tasks. The second is training stability and scaling behavior, because dynamic gating mechanisms often introduce additional optimization difficulties.

In addition, the 25MB state and 10 million token figures come from the paper's own report, and independent reproduction and third-party evaluation are yet to appear. For researchers and engineers focused on long-context modeling, Muir provides a new direction worth tracking: under fixed-state constraints, how to approach the effectiveness of full attention through more refined information allocation strategies.

(This article is based on publicly available Reddit community discussions and information from the paper abstract. For specific experimental details, please refer to the original paper.)

One-line Summary

Google Research's Muir uses sensitivity gating and matched-filter readout to carry a ten-million-token context with a 25MB state.

Paragraph Summary

The article focuses on Muir, a linear attention model proposed by Google Research. Addressing the contradiction between Transformer attention's quadratic memory growth with sequence length and linear attention's lossy compression with fixed states, Muir introduces two major mechanisms: sensitivity-allocated gating dynamically allocates limited state capacity based on the input's influence on the output, allowing key information to be prioritized for retention; matched-filter readout draws on signal processing ideas to precisely extract information from the compressed state in a way aligned with historical patterns. According to the paper abstract, the model stores a 10 million token context with about 25MB of state, decoupling memory usage from sequence length and offering deployment appeal for edge devices and limited-memory scenarios. The article also points out that fixed states are inevitably lossy, and the Reddit community is focused on its comparison with Mamba and RWKV, training stability, and the lack of third-party reproduction and validation of the metrics.