LeMario: An Open-Source Experiment in Training JEPA World Models with Super Mario Bros

Introduction: Understanding AI "World Models" Through a Game World

An open-source project called LeMario has recently caught the attention of the AI research community. It attempts to train a JEPA (Joint-Embedding Predictive Architecture)-based world model on the classic game Super Mario Bros, making it a fascinating effort to bring Yann LeCun's cutting-edge architectural ideas to life in a concrete, visualizable setting.

Why use a game from the 1980s for frontier AI research? Precisely because of its "simple yet complete" nature — it has clear physical rules, predictable dynamics, yet is complex enough to test whether a model truly "understands" how the world operates.

What Is a JEPA World Model

The Core Idea and Technical Origins of JEPA

JEPA was proposed and championed by Meta's Chief AI Scientist Yann LeCun as a self-supervised learning architecture. JEPA didn't emerge from thin air — it builds on years of LeCun's research into self-supervised learning and energy-based models. In 2022, LeCun published a 60-page paper, A Path Towards Autonomous Machine Intelligence, systematically laying out his vision for artificial general intelligence, with JEPA as one of its core technical modules. Its intellectual roots trace back to Predictive Coding theory (Hinton et al.) and contrastive learning methods like SimCLR and BYOL — all of which aim to help models learn the intrinsic structure of data without relying on labels. The Meta team has since released I-JEPA (image version) and V-JEPA (video version), validating the feasibility of representation-space prediction on multiple benchmarks. LeMario can be seen as extending this framework to interactive, dynamic environments.

Unlike mainstream generative models (such as GPT or diffusion models), JEPA does not aim to reconstruct or predict data at the pixel level. Instead, it performs predictions in an abstract embedding space.

This distinction is crucial. Traditional generative approaches must predict every pixel in a scene, forcing the model to spend enormous compute modeling details that are inherently unpredictable and irrelevant (like the random texture of clouds drifting in the background). JEPA instead predicts "what happens next" at a semantic level, allowing the model to focus on capturing the fundamental dynamics of the world rather than surface-level visual noise.

The Significance of World Models and the Research Landscape

A "World Model" refers to an AI's internal understanding of how an environment evolves over time and in response to actions. This concept has deep roots in reinforcement learning — in 2018, David Ha and Jürgen Schmidhuber's paper World Models systematically demonstrated how a VAE+RNN architecture could enable agents to learn in "dreams" without extensive interaction with the real environment. DeepMind's DreamerV1/V2/V3 series continued pushing this direction, achieving significant sample efficiency gains on tasks like Atari and DMControl. OpenAI's MuZero combined world models with tree-search planning to reach superhuman performance in board games and Atari.

A high-quality world model allows an agent to mentally simulate the consequences of an action before actually executing it in the real world — a key pathway toward efficient decision-making and higher-order intelligence. LeMario's JEPA-based approach differs fundamentally from the above: rather than reconstructing the future in pixel space or a decoded latent space, it predicts directly at the semantic embedding level, sidestepping the "hallucination" and high compute costs inherent to generative models.

LeCun has long argued that true AGI won't be achieved by stacking language models alone — it must include the ability to model the physical world. JEPA is one of his technical answers to that challenge.

LeMario: Why Super Mario Bros

An Ideal Experimental Sandbox with an Academic Tradition

Super Mario Bros actually has a long history as an AI research platform. As early as 2009, the Mario AI Championship attracted research teams worldwide to tackle level completion using reinforcement learning, A* search, and other methods. One of the precursors to OpenAI Gym included a Mario environment based on a NES emulator. From an academic standpoint, the core appeal of the Mario environment lies in its partial observability — players can only see the current screen, yet must anticipate enemies and obstacles off-screen. This closely mirrors real-world embodied intelligence scenarios and effectively tests a world model's forward prediction capabilities.

As an experimental platform, Mario also offers multiple natural advantages:

Clear rules: Physics like gravity, collision, and jump trajectories are fixed and reproducible;
Rich dynamics: Enemy movement, platform jumping, and item interactions provide sufficient complexity;
Intuitive visualization: Researchers can directly observe whether the model accurately predicts Mario's position and state in the next frame;
Controllable data: The game environment can generate unlimited training data without expensive manual annotation.

Applying JEPA to Mario is fundamentally about testing one question: can a model, purely through observing sequences of game frames, autonomously learn the evolutionary rules of the game world within an abstract representation space?

From Passive Observation to Active Understanding

The value of LeMario lies not in "completing the game" as an outcome, but in exploring whether a model can self-supervisedly learn the intrinsic dynamics of the game. If the model can accurately predict the abstract states corresponding to future frames within the representation space, it demonstrates that the model truly understands the logic of the Mario world — rather than merely memorizing visual patterns.

Practical Implications of This Technical Approach

Comparison with Generative Approaches

In recent years, Genie (Generative Interactive Environments), released by Google DeepMind in 2024, became a landmark result for the generative world model approach — based on a 1.1-billion-parameter video diffusion framework, it can generate interactive 2D game environments from a single image, stunning the research community. Meanwhile, GameNGen (running DOOM in real-time based on Stable Diffusion) and DIAMOND (Diffusion As a Model Of Actions and Noise) also demonstrated the potential of diffusion models as "neural game engines."

However, these systems commonly face three major challenges: weak physical consistency between frames (characters may "clip through walls"), extremely high real-time inference compute requirements, and frame "drift" caused by error accumulation over long sequences. The JEPA approach that LeMario represents offers a fundamentally different perspective: don't generate images, just understand structure. This approach has potential advantages in computational efficiency and robustness, making it especially suitable as a "cognitive foundation" for downstream reinforcement learning or planning tasks.

Implications for Embodied Intelligence and Robotics

The research value of world models extends far beyond gaming. On the industrial application side, Tesla's Autopilot team has publicly stated that its FSD system uses video prediction models internally to assist driving decisions. Humanoid robot companies like Figure and 1X are also actively integrating world models into motion planning pipelines. Research from Pieter Abbeel's team has shown that robots with high-quality world models require tens of times fewer real interaction samples when facing new environments — a "sample efficiency" advantage with enormous commercial value, since the cost of trial-and-error with real robots far exceeds that of virtual environments.

In embodied intelligence scenarios such as robotics and autonomous driving, enabling AI to mentally "rehearse" the outcome of an action before executing it is a core strategy for improving safety and sample efficiency. LeMario's exploration of validating JEPA in the controlled Mario environment accumulates valuable foundational experience for extending this architecture to more complex real-world tasks. Games are the lowest-cost, fastest-feedback testbed for world models, and successful methodologies can be incrementally transferred to high-value scenarios like industrial robotic arms and domestic service robots.

Conclusion: A Small Project, A Big Direction

Although LeMario remains a relatively niche experimental open-source project, the core question it addresses — how to make AI truly understand the world rather than merely imitate data — is one of the most important dividing lines in current AI research. From Yann LeCun's theoretical vision, to Meta's I-JEPA and V-JEPA, to community efforts like LeMario, a technical pathway from abstract theory to concrete application is gradually taking shape.

Starting from Super Mario Bros — a game that holds countless people's childhood memories — to explore the road toward next-generation AI architectures has a distinctly engineer-like romance to it. For developers who follow world models, self-supervised learning, and JEPA architecture, LeMario is a fascinating starting point worth tracking and reproducing.

It should be noted that due to limited publicly available information about this project, this article is largely an interpretation and extension based on the general technical background of JEPA and world models. For specific implementation details, readers are encouraged to consult the project source code and the author's original documentation.