What Comes After LLMs? Five Directions for Next-Generation AI Architecture

The Bottlenecks of Large Language Models Are Becoming Clear

Large language models (LLMs) have undeniably reshaped the entire AI industry. From ChatGPT to a wide range of open-source models, the paradigm built on Transformer architecture and next-token prediction has produced astonishing capability leaps over the past few years. Yet a growing number of researchers and engineers are coming to a sobering realization: relying solely on scaling next-token prediction is hitting a ceiling.

A Reddit user sparked a lively discussion in the tech community with a pointed question: "What comes after LLMs?" They accurately identified several core pain points with the current paradigm:

• High compute costs: Training and inference costs grow exponentially with parameter count, while marginal returns are diminishing.
• Hallucinations: Models confidently generate incorrect information in ways that are fundamentally difficult to eliminate.
• Frozen knowledge: A model's knowledge is "locked in" once training ends, with no natural mechanism for continuous updates.
• Weak real-world understanding: LLMs excel at manipulating linguistic symbols, but lack genuine modeling of causality, spatial relationships, and temporal dynamics in the physical world.

These problems cannot be solved simply by adding more parameters and more data. As a result, exploring next-generation AI architectures has become a shared focus across academia and industry.

Five Next-Generation AI Directions Drawing the Most Attention

World Models: Helping AI Truly Understand the Physical World

World models aim to give AI the ability to build an internal abstract representation of how the external world works — much like humans do. Unlike LLMs, which simply predict text sequences, world models emphasize modeling causal relationships, physical laws, and environmental dynamics.

The core logic here is that an agent should be able to simulate the consequences of its actions "in its head," enabling planning and reasoning. This is especially critical for scenarios that require interaction with the real physical world, such as robotics and autonomous driving. Many researchers believe that the absence of a world model is precisely why current LLMs' "understanding" remains superficial.

JEPA: Predicting Abstract Representations, Not Pixels

JEPA (Joint Embedding Predictive Architecture) is the direction Yann LeCun has most forcefully championed. Its central idea is to make predictions in an abstract representation space rather than reconstructing raw data space (such as pixels or tokens) point by point.

LeCun has repeatedly and publicly criticized the autoregressive LLM paradigm, arguing it cannot lead to genuine machine intelligence. JEPA attempts to have models learn the high-level structure of the world while ignoring unpredictable detail noise — bringing it closer to how humans actually learn. When we understand a scene, we don't need to memorize every individual pixel. This is widely considered an important technical pathway toward world models.

JEPA's architecture has two main components: a "context encoder" that processes observed inputs, and a "target encoder" that generates representations of the parts to be predicted — with a predictor operating in the embedding space between them. Unlike generative models (such as VAEs or diffusion models), JEPA does not need to reconstruct missing content at the pixel level, which naturally avoids overfitting to unpredictable details. Meta AI Research has already released concrete implementations like I-JEPA and V-JEPA, achieving competitive results on visual self-supervised learning tasks. Another potential advantage of this architecture is computational efficiency: since prediction targets are low-dimensional abstract representations rather than high-dimensional raw data, the training signal is more stable and theoretically requires less compute.

Mamba State Space Models: A New Architecture Challenging Transformers

If the first two directions focus on what to learn, Mamba focuses on what architecture to learn with. Mamba is based on State Space Models (SSMs) and aims to address the quadratic computational complexity that Transformers face when handling long sequences.

Mamba's advantage lies in its linear computational complexity with respect to sequence length, which delivers significant efficiency gains on long-sequence tasks such as long documents, genomic sequences, and audio. It represents a fundamentally different approach to sequence modeling than attention mechanisms. While it hasn't fully displaced Transformers yet, it provides an important reference point for AI architecture innovation.

The mathematical roots of SSMs come from control theory. The core idea is to compress historical sequence information into a set of latent variables (states) and update them recursively via matrix multiplication. Traditional SSMs use fixed state-transition matrices that are insensitive to input content. Mamba's key innovation is the introduction of a selective mechanism: state-transition parameters are dynamically generated based on the input, allowing the model to selectively remember or forget information — addressing the limited expressiveness of classical SSMs. During inference, Mamba can run in a purely recurrent mode with constant memory usage, making it highly attractive for deployment on resource-constrained devices. Mamba-2 and various hybrid architectures that combine Transformers with SSMs (such as Jamba) have already been released, exploring interleaved attention and SSM layers to balance performance and efficiency.

Continual Learning: Moving Beyond Frozen Knowledge

To address the pain point of frozen knowledge, continual learning (also called lifelong learning) aims to enable models to continuously absorb new knowledge after deployment without forgetting what they've already learned.

The central challenge here is catastrophic forgetting — when a model learns new tasks, it tends to overwrite knowledge from old ones. If this problem can be solved, AI systems would no longer require expensive full retraining and could instead grow incrementally, like humans. This would fundamentally change how models are iterated and maintained.

The root cause of catastrophic forgetting is that neural network parameters are globally shared — gradient updates for new tasks indiscriminately overwrite weights associated with old tasks. The main strategies for addressing this fall into three categories: regularization methods (such as EWC, which uses the Fisher information matrix to identify and protect important parameters); dynamic architecture methods (which dynamically allocate new network capacity for new tasks, as in Progressive Neural Networks); and replay methods (which store or generate samples from old tasks and mix them with new task data during training). In the context of large language models, parameter-efficient fine-tuning (PEFT) techniques like LoRA can also be viewed as a lightweight variant of continual learning. However, achieving truly robust continual learning over massive, heterogeneous data streams remains an open challenge — particularly in balancing forward transfer (new knowledge benefiting old tasks) against backward interference.

Neuro-Symbolic AI: Bridging Perception and Reasoning

Neuro-symbolic AI advocates combining the pattern recognition capabilities of neural networks with the logical reasoning capabilities of symbolic systems. Neural networks excel at perception and inductive generalization, but have weaknesses in strict logical reasoning, interpretability, and knowledge representation — while traditional symbolic systems are precisely the opposite.

Merging the two is believed to hold promise for simultaneously mitigating both hallucinations and insufficient reasoning ability. This "best of both worlds" approach is especially compelling in fields requiring high reliability and verifiability, such as mathematical proof and legal reasoning.

Neuro-symbolic AI is not a new concept — its history dates back to the 1990s — but it has regained attention in recent years as the reasoning limitations of LLMs have become more apparent. Representative fusion approaches include: feeding neural network outputs into symbolic reasoning engines (loose coupling); encoding logical rules as differentiable loss functions (as in Logic Tensor Networks); and having models directly generate executable programs or formal proofs (as in AlphaCode or Lean-assisted theorem proving). The "Chain-of-Thought" prompting technique in current LLMs is, in a sense, a soft simulation of symbolic reasoning steps. The true challenge lies in enabling end-to-end gradient flow between symbolic reasoning modules and neural perception modules, since symbolic operations are inherently non-differentiable — requiring reinforcement learning or relaxation approximations to work around this obstacle.

No Single Silver Bullet: How These Directions Complement Each Other

Interestingly, these five directions are not mutually exclusive competitors — they are more likely complementary and convergent. World models need good architectures to be realized; JEPA itself is a pathway toward world models; continual learning can be layered on top of any underlying architecture; and neuro-symbolic methods can inject reliable reasoning capabilities into any model.

Future AI systems will very likely be organic combinations of these ideas: a unified system with a world model, the ability to learn continuously, an efficient architecture, and integrated symbolic reasoning.

Perhaps There Are Overlooked Directions Too

The original post ended with an open question: "Are there any ideas people are overlooking?" This is worth pondering. Historically, major breakthroughs have often come from unexpected corners. Beyond the hot directions above, some relatively niche explorations also deserve attention:

• Embodied AI: Letting intelligence emerge naturally through interaction with real or simulated environments.
• Energy-Based Models: Another framework LeCun has expressed strong belief in.
• Architectures closer to biological brain mechanisms: Such as Spiking Neural Networks (SNNs).

Conclusion: From "Remembering More" to "Understanding Deeper"

The question "What comes after LLMs?" fundamentally reflects the entire AI field's anticipation — and anxiety — about the next paradigm shift. LLMs have demonstrated the power of scale, but also exposed the limits of scaling alone.

What is clear is that the central proposition of next-generation AI will shift from "remembering more" to "understanding deeper, learning more continuously, and reasoning more reliably". Whether the ultimate winner is world models, JEPA, Mamba, or some yet-unnamed combination, AI is standing at a new crossroads. For researchers and engineers, now may be the best time to place a bet on the next direction.