Latent Reasoning: The Next-Generation AI Reasoning Paradigm Beyond Chain of Thought

The Scale Trap of Chain of Thought

In the race to build reasoning capabilities in large language models (LLMs), Chain of Thought (CoT) was once seen as the key to unlocking complex reasoning. Formally proposed by the Google Brain team in 2022, the core idea is to guide models to generate step-by-step derivations before producing a final answer — significantly improving performance on tasks like mathematical reasoning and logical inference. It evolved into variants such as Zero-Shot CoT and Self-Consistency, becoming the standard paradigm for reasoning enhancement. But a deep discussion that went viral on Reddit raised a genuinely challenging point: we've started conflating "readable reasoning traces" with "actual computation." In one sentence: "Generating text is not the same as thinking."

This argument cuts to the heart of two critical pain points in today's reasoning models.

The first is the faithfulness problem: CoT-style reasoning traces may be decoupled from the actual computation the model performs. Faithfulness is a core concept in explainable AI — it asks whether the explanation a model generates truly reflects its internal decision-making process. Research from Anthropic, DeepMind, and others has shown that CoT reasoning chains exhibit a systematic pattern of "post-hoc rationalization" — models sometimes arrive at an answer first, then reverse-engineer plausible-looking reasoning steps. Stanford researchers found that injecting errors into key steps of a reasoning chain had far less impact on the final answer than expected. You might get a chain of seemingly sound steps that leads to a wrong answer, or a chaotic chain that somehow reaches the right conclusion. This means CoT reasoning traces are not a reliable audit trail.

The second is system cost. Autoregressive inference is the core mechanism of mainstream Transformer-based language models — the model generates one token at a time, appending it to the context before generating the next, making it an inherently serial process. This design creates serious efficiency bottlenecks at inference time: the longer the reasoning chain, the more KV Cache memory and attention computation grow. OpenAI's o1 model costs roughly 15x more per API call than GPT-4o, largely for this reason. When we ask a model to "think out loud," we're paying a steep computational price for readability.

Latent Reasoning: Keeping Thought Internal

A wave of cutting-edge research is driving an important shift: moving the inner loop of reasoning into latent space, only decoding to language at the final stage.

Latent space is the high-dimensional continuous vector space inside a deep learning model — the model's "internal language" for compressing representations of input data. Unlike text tokens, these continuous floating-point vectors are not constrained by a vocabulary and can encode far richer semantic information than discrete text. From an information-theoretic perspective, "translating" intermediate reasoning into natural language necessarily involves information loss — human language operates at a much coarser granularity than the model's internal computational precision. This is the core claim of latent reasoning: allow models to complete intermediate reasoning steps in high-dimensional continuous space, only executing the "decoding" — a lossy compression operation — at the final output stage. Several representative directions have emerged from this line of thinking.

Coconut: Continuous Latent Thought

Coconut (Chain of Continuous Thought) is a latent reasoning framework proposed by Meta AI Research in 2024. Its core technique introduces multiple rounds of "decode-free" forward propagation loops between the standard Transformer's embedding layer and output layer: the model performs multiple forward passes over the same context, feeding the output hidden states directly back as "input embeddings" for the next round — without going through softmax decoding into tokens. Experiments show that Coconut's latent thinking steps implicitly encode parallel exploration of multiple reasoning paths. The model completes multiple rounds of iterative reasoning internally before translating the result into human-readable language, avoiding the overhead of forcibly "translating" every intermediate computation step into text.

HRM: Separating Slow Planning from Fast Execution

HRM (Hierarchical Reasoning Model) and its text-based variants attempt to separate slow planning from fast recursive execution. This hierarchical structure closely mirrors the dual-process cognitive theory proposed by Nobel laureate Daniel Kahneman: a high-level planner (System 2 — slow, analytical) handles goal decomposition and strategy formation, while a low-level executor (System 1 — fast, intuitive) handles high-frequency processing of concrete steps. This design is not only more cognitively sensible but also more computationally efficient — the planning layer can run a larger model at lower frequency, while the execution layer uses a lightweight model for detail processing, enabling differentiated allocation of computational resources.

RecursiveMAS: Passing Latent Embeddings Between Agents

RecursiveMAS extends this idea to multi-agent systems: having agents pass latent embedding vectors to each other rather than verbose text messages. Current mainstream LLM-based multi-agent frameworks (such as AutoGen and CrewAI) universally use natural language as the communication medium, which has two inherent drawbacks: the bandwidth loss from compressing continuous internal states into discrete text, and the redundant encoding/decoding overhead on the receiving end. RecursiveMAS bypasses this by directly passing latent embedding vectors — analogous to the efficiency advantage of "shared memory" over "message passing" in distributed computing — dramatically reducing the communication cost of agent collaboration.

A valuable framework for understanding this shift is the distinction between language as interface vs. language as compute substrate. Language is undeniably essential for communication and abstraction, but forcing computational tasks like search and constraint solving to be serialized into text is both clunky and expensive. Language should be the interface for input and output — not the medium in which thinking itself happens.

The Black Box Wall: The Real Cost of Interpretability

Yet latent reasoning is not without its costs. If "thinking" happens across dozens of latent iteration cycles, we lose entirely the "observation window" that CoT once provided — even if that window was never very clear to begin with.

In high-stakes domains like healthcare, finance, and law, "zero visibility" is a genuine deployment barrier. The EU AI Act explicitly classifies scenarios such as medical diagnosis, credit assessment, and judicial assistance as "high-risk AI systems," requiring adequate transparency and explainability documentation. The Basel framework in finance and FDA guidelines for medical AI both require traceable model decision-making. When reasoning is entirely hidden in latent space, these compliance requirements run straight into a black box wall.

A constructive solution is to introduce an outer loop governance layer: use a symbolic or planning-based manager to construct an auditable directed acyclic graph (DAG) of sub-goals, with deterministic verification at each node — unit tests, constraint checks, rule validation, and so on. A DAG organizes sub-goals into a dependency graph with topological ordering, where each node represents an independently verifiable computation unit, allowing errors to be precisely localized. Its directed acyclic structure naturally prevents logical deadlocks from circular dependencies, and it describes parallel subtasks and conditional branching more naturally than a linear CoT chain.

This shifts the center of gravity for auditability from "reading the model's inner monologue" to "auditing the plan + checks + verified outputs." It's worth noting that regulators typically care about outcome auditability rather than process readability — outer-loop verification mechanisms provide independently verifiable objective evidence, not self-generated subjective statements from the model, and may actually be more compelling from a compliance standpoint than CoT. This represents a fundamental shift in thinking: from "trusting the internal process" to "verifying the external result."

BDH: Preserving Language Capability While Introducing Stateful Latent Computation

In this technical landscape, BDH (Dragon Hatchling) is a direction worth watching. Its distinguishing feature is that it attempts to preserve language modeling capabilities while introducing recursive or stateful latent computation — rather than being merely a supervised "puzzle solver."

According to the discussion, Pathway reportedly claims that a BDH-based system achieved 97.4% Top-1 accuracy on approximately 250,000 "Sudoku Extreme" puzzles, without using CoT and without answer backtracking. Sudoku is an effective diagnostic tool for evaluating constraint satisfaction problem (CSP) solving ability — from a computational complexity standpoint, Sudoku solving is NP-complete, and "extreme" Sudoku puzzles may require deep hypothesis-and-backtrack search that cannot be resolved through simple constraint propagation. A model that can directly output correct answers without CoT does suggest that some form of implicit constraint-solving mechanism has developed internally. But one should be clear-eyed: Sudoku is a closed-world problem with no commonsense reasoning, ambiguity resolution, or open-domain knowledge involved. High accuracy on this type of task cannot predict model performance on tasks requiring abstract analogy, counterfactual reasoning, or cross-domain integration — which are the core dimensions of general reasoning ability.

Depth Recurrence vs. Time Recurrence

Depth recurrence and time recurrence are two orthogonal dimensions in the design of latent reasoning systems. Depth recurrence refers to performing multiple rounds of iterative computation on the same input snapshot — similar to an iterative solver that refines repeatedly until convergence. Universal Transformer and DEQ (Deep Equilibrium Model) architectures are representative of this approach. Many recursive latent reasoners excel at this: iterating repeatedly over a fixed snapshot of a problem.

But real-world agent scenarios are fundamentally data streams — new tokens arrive continuously — which introduces the challenge of time recurrence: when do you advance the time step? Which states and memories do you carry forward? Traditional RNNs were designed for time recurrence but are limited by the vanishing gradient problem. Current large language models have largely abandoned time recurrence in favor of self-attention over the full context, at the cost of computational complexity scaling quadratically with sequence length. Mamba and other state space models (SSMs) are recent compromise attempts, but their reasoning ability still lags behind Transformers. This is a dimension that static problems like Sudoku don't touch. BDH's stated research direction is precisely to combine the two — high-bandwidth latent iteration and principled state/memory mechanisms that evolve over time — representing one of the core challenges of next-generation reasoning architectures.

BDH also offers a recoverable graph view along with sparse, localized state, providing a degree of native interpretability interfaces. It should be emphasized, however, that these interfaces complement system-level validation — they do not replace it.

Three Open Questions That Remain

This discussion ultimately converges on three core questions worth deep reflection across the entire AI community.

First, is CoT increasingly becoming an expensive interface artifact rather than a scalable reasoning path? From both cost and faithfulness perspectives, the answer seems to be trending toward yes. But CoT's value for interpretability and debugging is still difficult to fully replace in the near term.

Second, do high-stakes applications necessarily require a DAG/verification outer loop? Or can native model analysis interfaces meaningfully reduce governance burden? This question bears on the fundamental design direction of future AI system architectures, and directly affects compliance pathways for AI systems in heavily regulated domains.

Third, if latent recursion is the inner loop, what should the outer loop look like in practice? A DAG, unit tests, formal specifications, or a proof assistant? Formal verification tools such as TLA+ and Alloy are exploring integration with LLMs, bringing key DAG nodes within the scope of formal specification verification. Different domains may require different combinations of verification mechanisms.

From "Reading Thought" to "Verifying Results"

This discussion reveals a deep tension in LLM reasoning research: the fundamental trade-off between readability and efficiency, between interpretability and capability. Latent reasoning represents a more efficient path closer to "real computation," but at the cost of transparency.

The future answer is likely not a binary choice, but a layered architecture: latent space handles efficient inner-loop reasoning, a symbolic outer loop handles auditable planning and verification, and language returns to its proper role — as the interface for human-machine communication. This architecture already has engineering precedents: workflow orchestration tools like Apache Airflow describe verifiable computation pipelines with DAGs, complementing the neural computation of latent reasoning. Explorations like BDH — which balance language capability with stateful computation — as well as efforts like Coconut and HRM that bring depth recurrence into the reasoning inner loop, may be important steps toward this architecture.