The Post-Transformer Landscape: Comparing Mainstream Architectures in 2026

In 2026, the Transformer architecture's dominance is being challenged by a wave of emerging designs. From Mamba and RWKV to Titans and BDH, different labs have proposed radically different alternatives. A comparison table recently shared on Reddit systematically maps these architectures' core mechanisms, attention variants, and test-time memory capabilities, sketching out a technical panorama of the post-Transformer era.

Why Go Beyond the Transformer?

The Transformer's core bottleneck isn't expressiveness—it's resource efficiency. Standard attention's computational complexity grows quadratically with sequence length, and KV Cache memory overhead balloons in long-context scenarios. These constraints become especially acute when inference, memory, and continual learning are critical capabilities for AI systems.

The Reddit author's interest is clear: improving reasoning, memory, and adaptation "within practical compute and energy budgets." His judgment is that the next frontier may no longer rely purely on scaling, but on extracting more intelligence from limited resources. This perspective aligns closely with current industry discussions about diminishing returns from Scaling Laws, and explains why so many attempts are emerging simultaneously in directions like state space models (SSMs), linear attention, and hybrid architectures.

Divergence of Core Technical Paths

State Space Models and the RNN Revival

The Mamba series (from Gu, Dao, and others) follows the selective state space model path. Mamba-2 introduced the SSD framework to enable parallelized training, and Mamba-3 further adds complex-valued states and MIMO structure. RWKV-7 "Goose" represents the parallelizable RNN direction—it maintains constant memory usage and needs no KV Cache at all, giving it an inherent advantage in inference efficiency. xLSTM from NXAI reactivates the LSTM lineage with exponential gating and matrix memory (mLSTM).

What these paths share is: replacing full attention with recurrence or state space, maintaining constant memory overhead during inference. The trade-off is that on certain tasks requiring precise long-range retrieval, they may be less stable than full attention.

State Space Models (SSMs) originate from control theory, using a hidden state vector (h_t) to describe a system's evolution over time: (h_t = A h_{t-1} + B x_t), with output (y_t = C h_t). Their core advantage is linear time and constant memory—each step depends only on the previous step's state, with no need to store the full history. Early SSMs (like S4) use structured matrices (such as HiPPO initialization) to stably model long-range dependencies, but the coefficients (A, B, C) are input-independent, preventing the model from "selecting" what to remember based on content. Mamba's key innovation is making (B), (C), and the step size (\Delta) functions of the input (i.e., "selectivity"), thereby gaining content-aware gating capability while retaining linear complexity.

Mamba-2's proposed SSD (State Space Duality) framework reveals a deeper connection: SSMs with specific structure are mathematically equivalent to a masked form of linear attention, which allows training to be parallelized with matrix multiplication just like attention, rather than serially unrolled in RNN fashion. Understanding this is a prerequisite for understanding the later claim that "most SSMs use linear attention underneath." RWKV takes a different path: it trains with a parallelizable WKV operator along the time dimension, and degrades to a token-by-token recursive form during inference, so it naturally needs no KV Cache and its memory usage is independent of sequence length.

Engineering Trade-offs in Hybrid Architectures

More labs have chosen a pragmatic route—retaining some softmax attention layers while replacing the rest with linear attention or SSM layers. Qwen3-Next and Kimi Linear both adopt a 3:1 linear-to-attention ratio, while MiniMax-01 and Jamba push this ratio to 7:1 or 8:1. NVIDIA's Nemotron-H replaces almost all attention layers with Mamba-2.

The logic behind this hybrid strategy is straightforward: softmax attention handles precise retrieval in a few critical layers, while the remaining layers use more efficient mechanisms for most of the computation. MiniMax-01 and Jamba additionally layer MoE structures on top for another level of parameter efficiency optimization. Falcon-H1's design differs slightly, running attention heads and Mamba-2 heads in parallel within the same block rather than replacing them layer by layer.

Memory and Test-Time Adaptation

Titans (Google Research) introduces a neural long-term memory module that updates memory at test time based on "surprise." BDH (Dragon Hatchling, from Pathway) takes a more radical approach—using neuron graph structures to let attention emerge from local interactions, with a GPU implementation equivalent to ReLU low-rank plus linear attention. HRM (Sapient Intelligence) uses two coupled recurrent modules acting as a slow planner and fast executor respectively, completing reasoning in a single forward pass.

What these three share is the direction of giving models some form of "learning" capability during inference, rather than merely statically invoking weights fixed at training time. BDH explicitly draws on the neuroscience concept of Hebbian synapses, Titans uses the metaphor of a memory framework, and HRM is closer to an engineering implementation of dual-system theory from cognitive architecture.

Test-Time Memory / Test-Time Training differs from conventional inference in that the model modifies part of its own parameters or memory state during generation, rather than just updating the context cache. Traditional Transformers have frozen weights at inference, and all "adaptation" happens within the context window; methods like Titans maintain a writable and readable neural memory matrix, deciding write strength based on the novelty of input ("surprise," typically measured by gradient magnitude or prediction error). This amounts to doing a small-scale online learning step during inference.

Hebbian learning is a classic rule in neuroscience, often summarized as "fire together, wire together"—synaptic weights strengthen with the synchronous activation of pre- and post-synaptic neurons. BDH borrows this concept, using local interactions in neuron graph structures to let attention-like behavior emerge without explicitly defining Q/K/V. Dual-system theory (proposed by Kahneman) divides cognition into a fast, intuitive System 1 and a slow, reasoning System 2; HRM's two coupled recurrent modules are precisely the engineering mapping of this concept, with one handling slow planning and the other fast execution. Understanding these metaphors helps assess how radical each architecture's "test-time learning" is: Titans modifies memory state, BDH relies on structural emergence, HRM depends on module division of labor.

Architectural Overlap and Complementarity

The second dimension of the comparison table reveals that these architectures are not mutually exclusive. Most SSMs and linear attention models use linear attention mechanisms underneath—Mamba-2 via SSD structure, xLSTM via covariance updates. Titans instead retains softmax attention but layers a test-time memory module on top.

BDH is the only architecture touching all three dimensions simultaneously: linear attention, test-time weights, and neuroscience inspiration. Qwen3-Next and Kimi Linear both contain linear and softmax attention, and both use delta rules for test-time updates. LFM2 (Liquid AI) takes a unique hardware search route—its structure of 10 gated short convolution blocks plus 6 GQA blocks was discovered through hardware-in-the-loop search rather than human design.

StripedHyena 2 (Together AI / Arc Institute) combines three different scales of Hyena convolution operators (short, medium, long implicit) with attention, and its Evo 2 version focuses on genomics applications. Zamba2 (Zyphra)'s design is a Mamba-2 backbone with shared-weight attention blocks inserted every N layers, optimized specifically for parameter efficiency.

Real-World Engineering Constraints

A detail worth noting: several paths in the comparison table are labeled "inherited"—meaning their linear attention capability comes from constituent components rather than native design. Nemotron-H, Falcon-H1, and Zamba2 all fall into this category. This means their actual performance depends heavily on the maturity of the inherited components.

Another key variable is the introduction of hardware-in-the-loop search. LFM2's structure was not hand-designed by researchers, but found through a hardware-aware search process. This suggests future architecture innovation may increasingly be driven by automated tools rather than pure intuition.

Actual deployment choices often depend on specific scenarios: scenarios requiring extremely low inference latency and constant memory suit RWKV-7 or pure SSM architectures; scenarios requiring precise long-range retrieval still need to retain softmax attention layers; scenarios requiring continuous adaptation to new information may need solutions with test-time memory like Titans or BDH.

Hardware-in-the-Loop Search refers to feeding the target hardware's real performance metrics (latency, throughput, energy consumption, memory bandwidth) into the architecture or hyperparameter search loop, so that the search algorithm optimizes directly for the actual deployment environment rather than only proxy metrics like parameter count or perplexity. LFM2 uses this method to simultaneously determine layer type, count, and arrangement, so its structure often defies human design intuition.

This has two consequences: first, search results are highly dependent on the target hardware, and switching chips may require re-searching, reducing portability; second, architecture interpretability worsens, making it hard for researchers to explain "why 10 gated short convolutions plus 6 GQA blocks." In contrast, the "inherited" label points to another reality: many hybrid architectures' linear attention capabilities are not original but directly adopt existing components (such as Mamba-2's SSD or GLA), so their performance ceiling is constrained by the maturity of these components, and innovation is more reflected in combination ratios and inter-layer orchestration. This also explains why hybrid ratios (3:1 to 8:1) have become one of the most intensively experimented variables.

What to Watch Next

The Reddit author invited the community to add missing architectures and point out disagreements, which itself shows this field is far from converged. A few signals worth tracking: whether Mamba-3's complex-valued states can bring substantial improvements in language modeling; whether there is an optimal range for hybrid ratios; to what extent test-time memory mechanisms can replace traditional context window extension.

The more fundamental question is whether these architectures can truly outperform a well-tuned Transformer at equivalent compute. Most current comparisons remain at the mechanism level, lacking strictly controlled experiments. The answer will gradually become clear as these architectures enter larger-scale production validation.

One-line Summary

The Reddit comparison table maps the core mechanisms and practical trade-offs of post-Transformer architectures like Mamba, RWKV, and Titans.

Paragraph Summary

Based on an architecture comparison table shared on Reddit, this article systematically maps the main technical paths of the post-Transformer era. The Transformer's bottleneck lies in attention's quadratic complexity and KV Cache overhead, so SSM/RNN revivals (Mamba, RWKV, xLSTM), linear-softmax hybrid architectures (Qwen3-Next, Kimi Linear, Jamba), and test-time memory mechanisms (Titans, BDH, HRM) are emerging simultaneously. These architectures are not mutually exclusive; most SSMs can be reduced to linear attention underneath, and hybrid ratios are currently the most active experimental variable. Engineering-wise, hardware-in-the-loop search-driven design (LFM2) and a large pragmatic camp of "inherited components" have also appeared. The author cautions that current comparisons mostly remain at the mechanism level, lacking strictly controlled experiments, and whether they can truly outperform a tuned Transformer at equal compute remains an open question.