Kunlun Wanwei Matrix-Game 4.0: A Real-Time Video Model Game Running on a Single RTX 5090

Going from "watching AI-generated video" to "actually playing in an AI-generated world" is gated by one thing: real-time performance. Kunlun Wanwei's newly released Matrix-Game 4.0 claims to cross that threshold—it's the first video-model game to run on a single consumer-grade GPU, and it ships as open source.

A New Paradigm for Game World Models: The Engine Manages State, the Model Manages Pixels

The central architectural decision behind Matrix-Game 4.0 is splitting "how the game runs" and "how the game looks" into two parallel pipelines. The game engine maintains world state and interaction rules, while the video generation model handles only the real-time rendering of detailed visuals.

Concretely, a Coding Agent builds a lightweight game through code. The engine acts as the "referee," continuously maintaining world state, executing interaction rules, and outputting structured frames and dynamic semantic conditions. The video model acts as the "painter," turning simple geometry into rich imagery.

This division of labor addresses a problem that the previous generation of interactive world models never solved: the lack of an independent, verifiable mechanism for maintaining world state. Under the old approach, the persistent existence of scenes and objects depended on the model's internal representations and historical memory, so developers could not precisely read, modify, or verify the state of any individual object. Matrix-Game 4.0 "deposits" the world in an explicit engine rather than in the model's memory.

Hard Numbers: 480p/60FPS, Sub-150ms Latency

Matrix-Game 4.0 delivers a clear set of performance figures:

• On a single NVIDIA RTX 5090, the game runs at 60 FPS at 480p resolution
• End-to-end latency from player keypress to first frame response is under 150 milliseconds
• Steady-state generation speed for single-step denoising reaches 83 FPS
• Peak VRAM usage is 18.5 GiB

Compared with other approaches, Code World Model has latency of about 10.2 seconds, StreamRender-H3 about 545 milliseconds, and AgentGarten about 878 milliseconds, while Matrix-Game 4.0 comes in at just 112 milliseconds. In terms of compute requirements, the other approaches require 1× H200 141GB, 8× GB200, and 1× H100 datacenter cards respectively, whereas Matrix-Game 4.0 needs only a single consumer-grade GPU.

DiT (Diffusion Transformer) is the key technical term here. Traditional diffusion models mostly use U-Net convolutional structures, whereas DiT replaces the denoising backbone with a pure Transformer, using self-attention to process latent features cut into patches. It is naturally suited to multimodal conditioning—text, structured frames, and historical frames can all be concatenated as token sequences—and makes "causal" temporal modeling more natural: this solution uses a 30-layer causal DiT, where self-attention reads the K/V cache and cross-attention reads the text K/V, thereby reusing historical computation during frame-by-frame generation. The trade-off is high attention cost, which must be suppressed through KV caching and operator fusion—this is precisely the prerequisite for reaching real-time frame rates on a single card.

FP8 quantization and fused kernels are the other half of the low-latency story. FP8 refers to representing weights and activations with 8-bit floating point (E4M3/E5M2 and similar formats), halving memory footprint and bandwidth compared with FP16, and significantly improving matrix multiplication throughput on GPUs that support the precision, while the impact on diffusion model output quality is usually acceptable. But quantization alone does not automatically make things faster—if each layer launches a separate kernel, scheduling and memory access overhead will eat the gains. So the team wrote fused kernels for RoPE, normalization, and FP8 quantization, packing multiple element-wise operations into a single GPU call; they then used CUDA Graph to pre-record the entire DiT backbone into a graph that is replayed in one shot at runtime, eliminating per-kernel CPU launch overhead. Together, these let the text encoder, DiT, and VAE all stay resident on a single consumer-grade GPU.

Three Fundamental Bottlenecks: Why a World You Can "Walk Around In" Isn't Fun

In its release materials, Kunlun Wanwei lays out three structural problems with the previous generation of interactive world models, which form the starting point for Matrix-Game 4.0's design.

World state lacks an independent maintenance mechanism

Without an external engine maintaining explicit state, a model can "remember" images it has seen, but that does not mean developers can precisely read, modify, and verify the state of every object, nor is it easy to guarantee that characters and events outside the field of view always evolve according to specified rules.

Long-term interaction consistency is hard to guarantee reliably

Autoregressive generation depends on previous generation results, so local deviations can propagate and accumulate across continuous interaction. When a player explores for a long time and repeatedly returns to the same location, scene structure, object identity, and interaction outcomes may all drift. Expanding context and introducing memory mechanisms can improve the problem, but they do not provide game-grade long-term consistency guarantees.

Plausible visuals do not equal reliable rules

A model can generate visual effects such as opening doors, collisions, or attacks, but a complete game also requires precise numerical resolution—health changes, item ownership, quest progress, and win/loss determination. These mechanisms must be explicitly definable, modifiable, and testable by developers, and must remain consistent over long runs.

All three problems point to the same thing: what a game needs is a world that can be read, modified, and verified.

The Full Pipeline: Data, Training, and Inference Acceleration

A "game world" capable of running in real time rests on connecting three layers of capability: data, training, and runtime.

The data layer builds a structured-frame-to-detailed-frame cross-domain alignment dataset based on Unreal Engine 5, covering four major scenarios: the CitySample megacity, open desert, complex castle, and high-dynamic air combat. A self-developed lightweight geometric abstraction collection pipeline, while keeping camera trajectories and timestamps strictly synchronized, abstracts complex scenes into low-poly structural representations and aligns them pixel-by-pixel frame-by-frame with high-poly, cinematic-quality imagery.

The dynamic Caption mechanism lets descriptions follow the game. The system associates object IDs in the scene with semantic descriptions, retrieves currently visible objects and their real-time states from the game engine, and fills them into prompt templates at runtime. Structured frames provide position, layout, and motion information, while Captions supplement semantic information such as objects and states; together they constrain generation.

On inference acceleration, the team uses CUDA Graph to pre-record the DiT backbone computation and replay it in one shot at runtime; writes specialized fused GPU kernels for RoPE, normalization, and FP8 quantization; recalculates text-condition KVs only when the Caption updates; and switches video history KVs to a ring buffer. Ultimately, the text encoder, DiT, and VAE can all stay resident on a single RTX 5090.

The video model is adapted from Wan2.2 5B into video-to-video mode. The engine-rendered structured frames and the video to be generated are concatenated by channel (148-channel input), using a 30-layer causal DiT, where self-attention reads the K/V cache and cross-attention reads the text K/V.

Coding Agent: Rewriting Games with Natural Language

The most imaginative part of Matrix-Game 4.0 may be how easy it is to "mod." With the Coding Agent, players can customize their own game on top of existing game code using natural language: adjust terrain, place scene objects, modify NPC behavior and interaction logic, or even design entirely new gameplay.

For example, "make monsters swarm in from all directions" not only means more monsters appear on screen, but also requires implementing spawn positions, movement patterns, attack behavior, and win/loss conditions. Programming agents can also help define a scene's semantics and visual style: the same simplified building can be set as an old seaside house or a shop in a futuristic city.

Creativity no longer gets stuck in the 3D asset production pipeline; it can move directly into a "playable, iterable" stage.

Open-Source Strategy and Future Roadmap

Matrix-Game 4.0 is open-sourced at release, with code, weights, and white models all available, and the Hugging Face repository at Skywork/Matrix-Game-4.0. Compared with Code World Model and StreamRender-H3 being already open and AgentGarten being partially open, Matrix-Game 4.0 is more open.

The team says it will continue iterating on the model: deploying a refiner to improve image resolution, exploring multiplayer experiences that stay consistent across player viewpoints, providing finer-grained visual control, and expanding support for more game genres and scenarios.

From "controllable video" to "runnable world," the technical significance of this step is that it gives video generation models real value for local deployment and real-time interaction for the first time, rather than being an offline-rendered demo video. For small teams and individual creators, the cost of building a lightweight game engine is falling sharply, and "small teams can build worlds too" may no longer be just a slogan.

One-line Summary

Matrix-Game 4.0 uses a division of labor where the engine manages state and the model manages visuals to achieve a playable 480p/60FPS world on a single RTX 5090, and it's open source.

Paragraph Summary

Kunlun Wanwei has released and open-sourced Matrix-Game 4.0, which it calls the first video-model game to run on a single consumer-grade GPU: 480p/60FPS on an RTX 5090, end-to-end latency under 150 milliseconds, and peak VRAM of 18.5 GiB. Its core is separating game execution from visual presentation—the engine maintains world state and outputs structured frames, while the video model only handles real-time rendering, solving the three major bottlenecks of previous-generation world models: unverifiable state, long-term consistency drift, and unreliable rules. It is supported by a three-layer pipeline spanning data, training, and inference: a UE5 cross-domain alignment dataset, a dynamic Caption mechanism, and engineering optimizations such as CUDA Graph pre-recording, FP8 fused kernels, and ring-buffer KV caching. With the Coding Agent, players can rewrite gameplay in natural language, and it is open-sourced at release.