ReplaySSM: Rethinking State Storage to Accelerate Hybrid Model Decoding

Hybrid Models are becoming an important direction in the evolution of large language model architectures. By interleaving State Space Model (SSM) layers among a small number of attention layers, they substantially reduce computational overhead while preserving long-context capabilities. Mainstream models such as NVIDIA Nemotron 3, Qwen3.5 through Qwen3.8, Kimi-K3, and GLM-5.3-Flash have all adopted this paradigm.

However, hybrid models face an often-overlooked bottleneck during the decoding phase: the way SSM layer states are stored directly affects inference speed. ReplaySSM proposes a scheme that rethinks state storage strategy, with the goal of making hybrid model decoding more efficient.

The Hidden Bottleneck in Hybrid Model Decoding

During decoding, a traditional SSM layer only needs to maintain a fixed-size hidden state at each time step, which is theoretically more memory-efficient than the KV Cache of attention layers. But in a hybrid architecture, SSM layers and attention layers alternate, making state management and read/write paths more complex.

When the model needs to pass and restore states across different layers, frequent state access introduces additional overhead. This overhead is amplified in long-sequence generation scenarios and becomes a key factor constraining decoding throughput. This is precisely the core insight of ReplaySSM—the problem is not the computation of the SSM itself, but how states are stored and replayed.

State Space Models (SSMs) are a class of architectures that process sequences via linear recurrence, with representatives such as Mamba and Mamba-2. Unlike attention mechanisms, which need to perform pairwise comparisons across all historical tokens at each time step, SSMs compress historical information through a hidden state h_t that evolves over time: h_t = A·h_{t-1} + B·x_t, with the output obtained through a linear mapping of h_t. This means the computation per decoding step is independent of sequence length, and memory usage remains constant, making SSMs a natural substitute for attention layers in long-context scenarios. The trade-off is limited state capacity, which performs worse than attention on tasks requiring precise retrieval of distant details. Hybrid models attempt to compensate for this shortcoming with a small number of attention layers while using a large number of SSM layers to lower overall computation and memory costs—this is also the premise for understanding the motivation behind ReplaySSM's optimization.

The Core Idea of ReplaySSM

The core of ReplaySSM lies in redesigning the storage and restoration mechanism for SSM states. Rather than passively maintaining the full state at every decoding step, it reconstructs states on demand through replay.

The ingenuity of this approach is that it shifts state storage from "continuous maintenance" to "on-demand computation," seeking a better balance between storage cost and computation cost. For layers in a hybrid model whose states change slowly or predictably, the replay strategy can significantly reduce memory usage and the number of memory accesses.

It should be noted that publicly available technical details remain limited. The specific implementation of ReplaySSM—such as replay trigger conditions, state compression granularity, and how it coordinates with the attention layers' KV Cache—has yet to be further disclosed.

KV Cache and state maintenance during decoding is the background for understanding the differences between the two storage strategies. In autoregressive decoding, attention layers typically cache the Key and Value of all historical tokens (i.e., the KV Cache), whose memory grows linearly with sequence length, but each step only requires appending rather than recomputation; SSM layers, by contrast, retain only a fixed-size state. When the two are stacked alternately, a hybrid model often needs to manage both types of caches simultaneously and move states between layers. What ReplaySSM calls "replay" means that when a certain intermediate state is needed, it starts from an earlier checkpoint state and quickly recomputes several steps using known input tokens to restore it, thereby trading acceptable computation for a smaller resident memory footprint. This "computation for storage" trade-off is often more worthwhile in GPU decoding scenarios where memory bandwidth becomes the bottleneck, because recomputation can fully utilize compute power, while reduced state reads and writes directly alleviate bandwidth pressure.

Significance for the Hybrid Model Ecosystem

If ReplaySSM's method can be put into practice, its impact on the hybrid model ecosystem will be multifaceted.

Lower inference costs would make hybrid models more competitive on edge devices and in long-context scenarios. Optimizing state storage also means the same memory can support longer contexts or larger batch sizes. For model vendors that have already adopted hybrid architectures, such optimizations can be integrated directly into existing inference stacks without retraining models.

Competition in hybrid models is shifting from "architecture design" to "inference engineering." ReplaySSM represents a new class of optimization directions—not changing the model itself, but unlocking performance by rethinking state management.

One-line Summary

ReplaySSM reconstructs SSM state storage through on-demand replay to accelerate hybrid model decoding.

Paragraph Summary

Hybrid models interleave State Space Model (SSM) layers among attention layers to balance long context with low computational overhead, and have been adopted by Nemotron 3, the Qwen3.5 series, Kimi-K3, GLM-5.3-Flash, and others. But the article points out a hidden bottleneck in the decoding phase: SSM layers and attention layers alternate, making state management, read/write, and restoration paths complex, and the overhead from frequent state access is amplified in long-sequence generation. ReplaySSM's core idea is to change state storage from "continuous maintenance" to "on-demand replay," seeking a better balance between storage cost and computation cost, especially for layers whose states change slowly or predictably. If implemented, this approach can reduce inference memory usage, support longer contexts or larger batch sizes, and be integrated into existing inference stacks without training. The article also notes that its specific implementation details have not yet been fully disclosed.