Latent Lattice: Breaking Language Model Compute Bottlenecks with Persistent Inter-Layer States

Lattice of Latents expands computational capacity by adding per-layer persistent horizontal latent states, significantly reducing perplexity with unchanged parameters.
Lattice of Latents proposes a new architecture extension: instead of deepening layers or adding recurrent weights, it adds a set of horizontal latent states to each layer that recur along time and run parallel to the vertical residual stream. These latent states directly participate in attention, KV cache writes, and MLP operations, elevated to conditional signals on par with the main stream. With 406M parameters trained on 20B tokens, validation perplexity dropped from 15.41 to 14.40 with exact parameter match, and further to 13.61 when stacked with hyper-connections and input residuals, showing complementarity. It generally improves across architectures like Gated DeltaNet and Mamba-2, with small-state Lattice variants surpassing 4x larger state baselines. The authors identify persistent inter-layer states as an independent scaling axis, but inference latency impact and larger-scale performance remain to be verified.
The conventional path to increasing per-token compute in language models typically involves deepening the network, introducing recurrent layers, or adding latent reasoning steps. A new architecture called Lattice of Latents takes a different route: increasing the number of active computational states the model maintains.
From Residual Stream to Parallel Latent States
Traditional Transformers rely on a vertical residual stream running through the depth, with information passing level by level across layers. Improvements such as hyper-connections, mHC, attention residuals, or value residuals all essentially optimize this vertical pathway. Lattice of Latents takes a different approach: it adds a set of horizontal latent streams to each layer, which run recurrently along the time dimension and exist in parallel with the vertical residual stream.
The key point is that these latent states are not merely internal memory for sequence mixing as in hybrid Transformer-SSM architectures. They directly participate in attention computation of the vertical stream, KV cache writes, MLP operations, and decoding. In other words, latent states are elevated to conditional signals on par with the main computational stream, rather than passive intermediate variables.
Residual Stream is a core concept in the Transformer architecture, referring to an information pathway that runs through the entire network depth, formed by accumulating each layer's output through residual connections. In a standard Transformer, each layer's attention sublayer and feedforward sublayer add their computed results to this stream, so the residual stream can be viewed as a "shared workspace" through which the model passes and accumulates information across different levels of abstraction. Improvements such as Hyper-Connections enhance the expressiveness of this pathway by introducing multiple parallel residual streams or learnable connection weights, but they still essentially optimize vertical information flow. Understanding the role of the residual stream helps grasp the differentiated design of Lattice of Latents: it does not replace or optimize the vertical residual stream, but rather adds a set of horizontal, cross-token persistent latent state streams alongside it, giving the model additional parallel computational states beyond depth.
KV Cache is a key mechanism during autoregressive language model inference. When generating each new token, the model needs to compute attention, which requires the current token's query vector to interact with the keys (Key) and values (Value) vectors of all historical tokens. To avoid recomputing historical tokens' keys and values, the inference system caches them—this is the KV cache. The size of the KV cache grows linearly with sequence length and is one of the main memory bottlenecks in long-context inference. Lattice of Latents has latent states directly participate in KV cache writes, meaning these horizontal latent streams not only affect the current layer's attention computation but also inject information into the cache accessible to subsequent tokens. This is fundamentally different from SSM states that serve only as internal memory for sequence mixing: the latter typically do not enter the attention key-value space, whereas Lattice's latent states become conditional signals that the attention mechanism can directly read.
Experimental Data: Perplexity Drops Across the Board with Unchanged Parameters
In a Transformer trained on 20B FineWeb-Edu tokens, the 406M-parameter Lattice model reduced validation perplexity from 15.41 to 14.40, with parameter count exactly matching the baseline. This gain is reproducible across multiple architecture variants.
More noteworthy is the stacking effect with existing residual enhancement techniques. When combined with hyper-connections and input residuals, the baseline Transformer's perplexity was 15.41, which became 14.22 with the same enhancements, and further dropped to 13.61 when Lattice was added. This shows that the computational capacity provided by Lattice and the optimization of residual pathways are complementary rather than substitutive.
On Gated DeltaNet, Mamba-2, Triadic GDN, and the GDN-Mamba-2 hybrid architecture, Lattice improved perplexity at every tested native state size. The Lattice-GDN and Lattice-Triadic-GDN variants even outperformed pure baseline models with 4x larger native states.
Gated DeltaNet, Mamba-2, and Triadic GDN are linear attention and state space model variants that have emerged in recent years. Mamba-2 is based on selective state space models (SSM), achieving efficient sequence modeling through input-dependent state transitions, with a hidden state recursively updated along the time dimension at its core. Gated DeltaNet belongs to the linear attention family, using the Delta rule (an update rule based on prediction error) combined with gating mechanisms to maintain a matrix-type memory state. Triadic GDN is an extension of DeltaNet that introduces triadic interactions to enhance state update expressiveness. A common feature of these architectures is maintaining a native state for sequence mixing, with state size being an important capacity hyperparameter. Lattice of Latents generally improves perplexity on these architectures, and small-state Lattice variants can surpass pure baselines with 4x larger states, indicating that the capacity provided by horizontal latent states complements rather than simply replaces native sequence mixing states.
Persistent Inter-Layer States: A New Dimension for Language Model Scaling
These results point to a clear conclusion: persistent per-layer states can serve as an independent axis for scaling language model computation. In the past, we were accustomed to working on depth (more layers), width (larger hidden dimensions), or recurrence (shared layer weights), but Lattice suggests that computational states flowing horizontally between layers and persisting across tokens constitute an underestimated source of capacity in itself.
This approach is fundamentally different from recurrent architectures. Recurrent layers share weights and unfold time along depth, whereas Lattice's latent states are independent per layer and recur along time. It is also not equivalent to SSM memory mechanisms, because latent states directly participate in conditioning attention and MLPs rather than serving only sequence mixing.
From an engineering perspective, this design may introduce new trade-offs between memory and computation. Latent states need to persist across tokens, implying additional cache overhead, but the performance gain under unchanged parameter count may be worthwhile in inference scenarios. The paper does not discuss the specific impact on inference latency, which will be a direction requiring further verification.
The paper's authors conclude that these results identify persistent inter-layer states as a complementary axis for scaling language model computation. For researchers exploring architectures beyond Transformers, Lattice of Latents provides a modular component that can be combined with existing residual enhancements, linear attention, and state space models.

Perplexity is the most commonly used intrinsic metric in language model evaluation, defined as the exponential of the model's negative log-likelihood over the validation set token sequence. Lower perplexity indicates that the model is more confident and accurate in predicting real text. Given a fixed tokenizer and validation set, a 1-point perplexity drop (e.g., from 15.41 to 14.40) typically implies a substantive improvement in the quality of the model's probability assignments, but the absolute significance of this improvement must be judged in the context of training scale. A training scale of 20B tokens is on the smaller-to-medium side, and whether the gains observed at this scale hold for larger models and more data is key to evaluating the value of the architectural innovation. Moreover, perplexity improvement does not directly equate to downstream task performance gains, so the paper's conclusions still need further validation on tasks such as GLUE, SuperGLUE, and reasoning benchmarks.
Positioning from Multiple Perspectives
According to the paper abstract from the rss source, Lattice's core innovation lies in elevating latent states from "internal memory" to "conditional signals." This formulation contrasts with the design philosophy of current hybrid architectures (such as Mamba combined with Transformer). In hybrid architectures, SSM states primarily handle sequence mixing responsibilities; whereas Lattice's latent states directly write to the KV cache and participate in MLP computation, with a role closer to a parallel, cross-layer persistent computational workspace.
Currently, this work has been released as a paper, with no independent third-party reproduction or community discussion as corroboration. Its gain magnitude (about 1 point perplexity reduction) is statistically significant at the 20B token scale, but performance on larger-scale models and more downstream tasks still needs to be observed. For readers interested in architectural innovation, Lattice's modular nature makes it worth adding to the experiment checklist.
One-line Summary
Lattice of Latents expands computational capacity by adding per-layer persistent horizontal latent states, significantly reducing perplexity with unchanged parameter count.
Paragraph Summary
Lattice of Latents proposes a new path for extending language model architecture: instead of deepening layers or introducing recurrent weights, it adds a set of horizontal latent states to each layer that recur along time and run in parallel with the vertical residual stream. These latent states directly participate in attention computation, KV cache writes, and MLP operations, and are elevated to conditional signals on par with the main computational stream. With 406M parameters trained on 20B tokens, validation perplexity dropped from 15.41 to 14.40 with an exact parameter match; stacking with hyper-connections and input residuals further reduced it to 13.61, demonstrating complementarity. It generally improves on architectures like Gated DeltaNet and Mamba-2, with small-state Lattice variants surpassing baselines with 4x larger states. The authors identify persistent inter-layer states as an independent axis for scaling language model computation, but the impact on inference latency and performance at larger scales remain to be verified.
Related articles

Abnormal Error Rate Spike in Claude Mythos 5.2 and Fable 5.2
Claude Mythos 5.2 and Claude Fable 5.2 show an abnormal spike in request error rates, marked as investigating on the official status page. This article covers the incident overview, practical impact on developers, and recommended responses.

ReAct Runtime Context Trimming: Why Claude Code Acts at 70%
A deep dive into ReAct runtime context trimming: why Claude Code compresses at ~70% usage, why tool calls dominate, why messages must be dropped in pairs, and why codebases don't need RAG.

AI Agent vs LLM: How Cursor and Claude Code Actually Work
An in-depth analysis of the essential difference between AI Agents and LLMs: LLMs generate code suggestions, while Agents modify files and run tests via tool calls. Using a Spring Boot bug fix as an example, it breaks down the workflows, permission control, and verification mechanisms of Cursor and Claude Code.