Injecting Physical Intuition into LLMs via JEPA World Models: Technical Pathways and Key Challenges

The LLM Dilemma: Language Models Trapped in "Mary's Room"

Large Language Models (LLMs) can fluently describe physical phenomena — they know that "apples fall," that "gravity causes acceleration," and can even recite the formulas for Newton's laws. But do they truly understand physics?

A Reddit user in the machine learning community raised a thought-provoking philosophical point: what LLMs learn is merely statistical correlations between tokens — for instance, the words "falls" and "gravity" frequently co-occur — rather than genuine physical intuition. This is precisely the classic "Mary's Room" thought experiment — Mary knows every physical fact about color, yet has never actually seen a color.

This thought experiment was proposed by philosopher Frank Jackson in 1982 and remains one of the most famous arguments in the philosophy of consciousness. Mary is a color scientist who has spent her entire life in a black-and-white room, learning everything about color vision through a black-and-white screen — all the physics and neuroscience there is to know. When she steps outside and sees red for the first time, does she learn something new? Jackson originally used this experiment to argue against physicalism — that even with all physical facts in hand, the subjective "qualia" of experience is still missing. In the AI context, this experiment has been revived to ask: can a statistical model that has never interacted with the physical world truly understand the world through text alone?

"LLMs are Mary."

This statement strikes precisely at the Achilles' heel of current large models: the chasm between propositional knowledge and grounded understanding. Propositional knowledge refers to factual knowledge expressible as linguistic propositions, such as "water boils at 100°C." Grounded understanding, on the other hand, emphasizes that knowledge must be bound to perceptual-motor experience to be truly grasped. This distinction traces back to philosopher Gilbert Ryle's classic division between "knowing that" and "knowing how," and aligns with the core claim of the embodied cognition school in cognitive science — that abstract thought fundamentally depends on patterns of interaction between body and environment. A model can recite physical laws but cannot internally "run" a physical process — this is an intuitive manifestation of that gap.

The Core Proposal: Training Embodied Intuition in Physics Simulations via JEPA

The author's proposed approach unfolds in three steps, with remarkably clear logic.

Step 1: Train a JEPA Model in a Physics Simulation Environment

Imagine training a JEPA-style (Joint-Embedding Predictive Architecture) model in a physics simulation environment (such as MuJoCo or a simple 2D world).

JEPA is an architectural paradigm systematically articulated by Yann LeCun in his 2022 paper A Path Towards Autonomous Machine Intelligence. Unlike generative models (such as the GPT series predicting the next token, or diffusion models predicting pixels), JEPA's core idea is to make predictions in an abstract embedding space. Specifically, it consists of an encoder that maps inputs into a representation space, and a predictor that predicts future state representations within that space. This design avoids the computational waste of pixel-level prediction — the model doesn't need to predict the exact value of every pixel, but only needs to capture semantic-level changes. LeCun argues this more closely resembles how the human brain works: we don't simulate visual scenes pixel by pixel in our minds; instead, we predict "what happens next" at an abstract level.

Unlike predicting pixels or tokens, a JEPA model predicts representations of future states in an abstract embedding space.

MuJoCo (Multi-Joint dynamics with Contact), as a representative simulation environment, is a high-fidelity physics engine developed by Emanuel Todorov and open-sourced after being acquired by DeepMind in 2021. It is specifically optimized for multi-joint contact dynamics, capable of efficiently simulating rigid body collisions, friction, tendon-driven actuation, and other complex physical processes. It serves as the default physics backend for classic control tasks in OpenAI Gym/Gymnasium (such as Hopper, Walker, Humanoid), and its high simulation accuracy and computational efficiency make it an ideal platform for training embodied intelligent agents.

The key lies in the "unforgiving" nature of the loss function: if the model's understanding of physics is wrong, the predictor will fail. The author argues that this constraint is far more stringent than "next token prediction" — language models can get by on grammar and statistics, but physics predictions are simply right or wrong.

Step 2: Let Physical Structure Emerge Naturally from JEPA Representations

Under such training pressure, the emergent embedding space should encode genuine physical structure: object permanence, momentum, and motion trajectories. Because only by capturing these underlying principles can prediction succeed. The model won't memorize surface textures; instead, it's forced to abstract the essential laws of "what makes prediction possible."

Object permanence is a developmental psychology concept proposed by Jean Piaget, referring to a cognitive ability that infants typically acquire around 8-12 months — understanding that objects continue to exist even when they are out of sight. This seemingly simple ability is actually the foundation for spatial reasoning and physical causal inference. For AI models, learning object permanence means being able to maintain tracking of an object's state (position, velocity, existence) even after it becomes occluded, which is a necessary prerequisite for any meaningful physical prediction.

Step 3: Freeze the Physical Representations and Connect Them to an LLM as a Conditioning Signal

Finally, freeze these learned physical representations and attach them as a conditioning signal to an LLM-style reasoning model. This way, the LLM would possess two capabilities simultaneously:

• Language-level physics knowledge (which it already has)
• Embodied physical intuition that can actually be "run forward" (newly added)

In the author's words, this makes physical knowledge more like a "computational primitive" rather than just a propositional fact. The core hypothesis is that downstream learning will accelerate significantly — the LLM doesn't need to rediscover that "objects fall" because this is already encoded in its representations.

Adjacent Research: Insights from V-JEPA and DreamerV3

This idea didn't emerge from thin air, and the author honestly pointed to several existing adjacent research directions.

V-JEPA (Video Joint-Embedding Predictive Architecture), released by Meta AI in 2024, is a concrete implementation of the JEPA philosophy in the domain of video understanding. It trains models to predict the abstract features of masked regions in representation space by masking large spatiotemporal regions of video (up to 90%). Unlike pixel reconstruction methods such as MAE (Masked Autoencoders), V-JEPA performs no pixel-level decoding at all — it learns by comparing the predicted representations of masked regions against their ground-truth representations. Experiments show that V-JEPA's learned representations perform excellently on action recognition and video understanding tasks, and can be trained via self-supervision without any labeled data, validating that prediction in representation space can efficiently capture physical motion patterns in video. The world model roadmap long championed by Yann LeCun has this very idea of "prediction in representation space" at its core.

DreamerV3, proposed by Danijar Hafner et al. in 2023, is a milestone in Model-Based Reinforcement Learning. At its core is a latent world model composed of three key components: an encoder that compresses high-dimensional observations into compact latent states, a transition model that predicts state transitions in latent space, and a decoder that maps latent states back to interpretable outputs. The agent performs extensive imagination rollouts in this "imagined" latent space, learning policies without needing to interact with the real environment. DreamerV3 was the first to achieve strong performance with a unified algorithm across more than 150 different tasks (including the Minecraft diamond challenge), demonstrating the feasibility and sample efficiency advantages of planning and predicting in abstract state spaces.

However, the author astutely observed: the specific combination of "JEPA-style prediction + simulation-grounded physical representations + LLM integration" doesn't seem to have been cleanly executed by anyone yet. This was precisely the motivation for posting to the community for verification.

Three Unresolved Technical Challenges

The author posed three technical challenges to the community, each striking at the heart of what it would take to make this approach work in practice.

Challenge 1: Does Fully Corresponding Prior Work Exist?

This is the most basic literature survey question. While there is extensive research in world models, embodied intelligence, and multimodal alignment, whether the precise recipe of "simulation-trained JEPA physical representations directly conditioning an LLM" has been validated remains to be confirmed. Notable related but not exactly corresponding works include: PaLM-E embedding robot sensor data directly into a language model, RT-2 converting vision-language model outputs directly into robot actions, and explorations with universal world simulators like UniSim. Each of these touches on some facet of fusing "physical understanding" with "language capability," but none represents the exact approach of directly conditioning an LLM with JEPA physical representations.

Challenge 2: How Should the Interface Between Physical Representations and Language Models Be Designed?

This is the core engineering challenge. The author listed several possible fusion approaches:

• Direct concatenation to prompt embeddings: The simplest approach, but potentially low in information utilization efficiency
• Cross-attention: More flexible, similar to the approach used in multimodal models like Flamingo, allowing language tokens to dynamically query physical representations

Flamingo, proposed by DeepMind in 2022, was a pioneering work in multimodal large models. Its key innovation was using a Perceiver Resampler to compress variable-length visual features into a fixed number of visual tokens, then inserting them between the layers of a frozen language model via cross-attention layers (gated xattn-dense layers). In the cross-attention mechanism, language tokens serve as queries while visual representations serve as keys and values, allowing each language token to dynamically "query" the most relevant visual information. The advantage of this design is that the language model's original capabilities are preserved (weights frozen), while deep fusion between vision and language is achieved through the newly added lightweight cross-attention layers. Subsequent models like LLaVA and Qwen-VL have adopted or refined this paradigm.

Based on recent experience with multimodal fusion, cross-attention tends to outperform simple concatenation because it preserves alignment flexibility between modalities.

Challenge 3: Will the Sim-to-Real Gap Destroy Transfer Effectiveness?

This is the most critical risk. Sim-to-Real transfer is a core challenge in robotics and embodied intelligence, also known as the "reality gap." No matter how precise a simulation environment is, it cannot perfectly reproduce every detail of the real world: minor deviations in friction coefficients, non-Gaussian distributions of sensor noise, complex deformations of flexible materials, dynamic changes in lighting conditions — all of these can cause policies that perform excellently in simulation to completely fail during real-world deployment. The field has developed multiple coping strategies, including Domain Randomization (randomly perturbing simulation parameters during training to enhance robustness), Domain Adaptation (aligning simulated and real distributions through adversarial training and similar methods), and System Identification (precisely calibrating simulation parameters to approximate real physics). OpenAI's 2019 work training a robotic hand to solve a Rubik's Cube using domain randomization is a classic success case of sim-to-real transfer.

But the author offers an intriguing defense: if the representations are abstract enough — encoding only universal principles like momentum, trajectories, and permanence — they might survive the sim-to-real gap. After all, gravity and conservation of momentum are isomorphic between simulation and reality. The strength of this argument lies in distinguishing between two types of knowledge: fragile knowledge that depends on specific simulation parameters (such as sliding distance under a particular friction coefficient) and abstract physical principles that are invariant across domains. If JEPA's training pressure is strong enough to force the model to retain only the latter, then transfer robustness can be expected.

Deeper Significance: Unifying Symbolic and Embodied Intelligence

Beyond specific implementation details, this vision touches on a fundamental tension in the current AI landscape: the unification of symbolic/linguistic intelligence and embodied/physical intelligence.

LLMs represent the pinnacle of the former, while JEPA and world models represent the exploratory path of the latter. Stitching the two together could theoretically enable AI that can both "speak" and "act," possessing both broad knowledge and deep intuition. This aligns with industry discussions that "world models are a key path to AGI." The backdrop to this discussion is that current AI development has clearly taken a "walking on two legs" approach — one leg being the pure language route represented by GPT-4, Claude, and others, growing increasingly powerful at text reasoning but lacking physical perception; the other being the perception-action route represented by robotics and embodied intelligence, possessing physical interaction capabilities but lacking high-level language reasoning. The convergence of these two paths is widely considered a necessary step toward more general intelligence.

That said, this proposal also faces hard practical questions:

• Composability of representations: Can physical intuition truly be compressed into a freezable, static embedding? Real reasoning often requires dynamic simulation rather than lookup. The essence of this problem is that physical reasoning is a procedural computation requiring unrolling over time steps, while frozen representations are more akin to declarative knowledge. Whether a finite-dimensional static vector can encode dynamical processes that inherently require iterative unrolling remains theoretically unsettled.
• The evaluation challenge: How can one prove that the LLM has acquired "genuine physical intuition" rather than just another layer of statistical correlation? This requires carefully designed out-of-distribution (OOD) tests. OOD tests examine generalization by deliberately constructing scenarios that differ from the training distribution — for example, testing with object shapes, mass combinations, or physical interaction patterns never seen during training. Only robust performance under OOD settings can provide initial support for the claim that "the model has learned real physical laws."
• Scale matching: Whether the complexity of simulation environments can support learning general physical intuition remains an open question. Even the most advanced physics simulation engines today still face accuracy and efficiency bottlenecks when simulating complex physical phenomena such as fluids, cloth, and granular materials, and the physical complexity of the real world far exceeds the coverage of any single simulator.

Conclusion: A Research Hypothesis Worth Prototyping

The author's final question to the community was: "Is this worth building a small prototype for?" From a technical standpoint, the answer leans toward yes. Even if the transfer effects prove limited, such a controlled experiment could reveal the boundaries of fusing linguistic and physical representations — and that alone would constitute a valuable negative result. In machine learning research, negative results are often underappreciated, but they serve an irreplaceable role in defining the applicable boundaries of methods and preventing the community from repeatedly stepping into the same pitfalls.

At a time when "next token prediction" is increasingly revealing its ceiling, seeking embodied, grounded supplementary signals for LLMs is becoming a consensus direction among more and more researchers. From Meta's V-JEPA to Google's PaLM-E and RT-2, from Yann LeCun's world model blueprint to various robotics labs' explorations of embodied large models, this trend is moving from academic discourse to engineering practice. This Reddit discussion is a vivid microcosm of that intellectual current at the community level.