8 AI Agents Building Games Together: How VibeGame Tackles AI Programming's Hardest Problem

Type a single sentence — "Make a Sekiro-style parry boss fight with a posture meter and deathblows" — and within tens of minutes, a playable 2D browser game appears: parries with real timing windows, deathblows triggered when the posture bar fills, and a boss whose heavy attacks have clear wind-up animations. This isn't a concept demo video. It's a system called What Game? (referred to in the paper as Prompt-to-Game development), open-sourced by a team from Nanjing University and Nanyang Technological University.

The input is natural language plus an optional reference image. The output is a complete, editable 2D game project. The team calls their approach adversarial development: 8 AI roles collaborate while actively challenging each other's work, and the game isn't delivered until it passes final review. This article breaks down the system's three-layer design — and along the way, explains why games are one of the hardest things for AI to build.

Why Games Are Especially Hard for AI

The paper opens with a direct statement: among all software, games are probably the most demanding. They must sustain an interactive world in real time, with visual coherence, balanced rules, and responsive feel. Neither of the two existing AI approaches can reach this bar.

The world-model route (systems like Genie) can learn interactive experiences from video, but what it produces is an implicit representation inside a neural network — no source code, no project structure. If you want to tune the jump height, there's nothing to precisely edit. The other route connects large models to commercial engines like Unreal, but those engines were designed for humans: project state is scattered across graphical interfaces and binary assets, and an agent can't see the full picture.

Verification is another problem. A game can be structurally valid and still crash at runtime, because player input, physics, animation, and game logic interact at millisecond timescales. Games are real-time systems, and agents reason orders of magnitude slower — by the time you read the logs, the character has died eight times.

There's also a widely overlooked issue: existing systems treat "first playable version" as the finish line, but in real development, delivery is where things begin. Users ask for new mechanics, visual changes, balance tweaks. VibeGame has a dedicated solution for each of these three problems.

Layer One: An AI-Native Game Engine

The foundation is Phaser, a JavaScript 2D web game framework whose projects are naturally all text files. But the team didn't stop at "plug an agent into an existing engine" — they defined three properties of an AI-native engine on top of Phaser.

No GUI dependency: The project is organized into a three-layer structure of scenes, nodes, and assets, all typed JSON text. Schema validation catches invalid references before runtime — for example, a reference to a nonexistent image is caught early rather than blowing up when the game runs.

Runtime accessibility: This is the cleverest piece. The engine exposes a programmatic interface for synchronous control: agents can pause the game, step forward one frame, fast-forward 50 frames, read and write node positions and velocities, inject semantic actions like "press jump," and take screenshots at any time. In short, it turns an asynchronous real-time system into a synchronously verifiable process — giving the agent a game console with a pause button, at its own pace.

Source-readable: The engine's full source code is copied into the project directory on initialization. When documentation is unclear, the agent reads the source directly. Documentation goes stale; source code doesn't. On the human side, the project includes a web workbench with four views: chat, asset management, object configuration, and direct play.

Phaser is an open-source 2D game framework built on HTML5 Canvas/WebGL, written in JavaScript, with a mature ecosystem and extensive community plugins and examples. The core reason for choosing it over Unity or Godot is its "all-text project structure": Unity projects contain large amounts of binary .meta files and scene serialization formats; Godot is closer to text but still has node configurations heavily tied to its graphical editor. A Phaser project essentially only needs HTML, JavaScript, and JSON — the entire project can be read and written by a language model directly, without any intermediate conversion layer. This is critical in AI-assisted development: when a model needs to understand "the current game state," it can scan all files in the directory directly, rather than relying on secondhand information from screenshots or logs.

Layer Two: A Team That Actively Challenges Itself

Eight roles with clear responsibilities: a Commander coordinates the overall process; a Designer expands ideas into a game design document; an Artist uses image generation models to produce concept art and assets; an Architect plans the structure; a Programmer writes code; an Auditor performs static checks; a Playtester runs live tests; and a Reviewer handles final approval.

The workflow has three phases. In the intent alignment phase, the Commander asks clarifying questions first; the Designer and Artist work in parallel to produce a design document and concept art for user confirmation; the confirmed document becomes the baseline for all subsequent development and evaluation. In the parallel development phase, art, design, and code workflows run simultaneously, with complex tasks broken into a four-role pipeline: Architect plans, Programmer implements, Auditor does static review, Playtester verifies live. In the adversarial correction phase, the Reviewer examines the complete project across functionality, visuals, and playability; any failure converts into a fix task that loops back until it passes.

The key design insight: the builders and the critics are two separate groups. The Auditor, Playtester, and Reviewer produce no code — they exist purely to find problems. The logic is simple: students shouldn't grade their own exams. A real case on the official workbench: the Auditor found that the damage window on the boss's ground slam fired two frames before the warning animation ended — by the time the player saw the impact indicator and thought they were dodging, they'd already been hit. The issue was sent back, the Programmer fixed it, it passed re-review, and the Playtester ran 12 scenarios before delivery. A two-frame timing bug that human eyes would almost never catch.

Model assignment also matters: in the default configuration, the Commander, Artist, and Architect use the most capable flagship models; the Designer and Programmer use mid-tier models; the Auditor uses a lightweight model; the Playtester and final Reviewer use models from a different vendor — expensive models serve as the brain and judge, cheaper ones run the pipeline, and two vendors are used in combination. The report honestly notes that GLM and MiniMax were tested but couldn't handle roles that require visual perception of the screen.

The "role separation" in multi-agent systems isn't VibeGame's invention, but its implementation in game generation is distinctive. In common single-model dialogue flows, the same model both generates and checks code — equivalent to a programmer doing their own code review, which software engineering practice has shown is prone to confirmation bias. Frameworks like AutoGen and CrewAI provide role-division scaffolding, but most don't include built-in programmable runtime inspection interfaces for real-time game validation. The reason VibeGame's Playtester can detect two-frame timing discrepancies is precisely the engine's frame-by-frame step API — a direct benefit of designing the runtime environment as a "programmable test bench," not of model reasoning capability alone.

Layer Three: Making Each Game the Starting Point for the Next

The third layer is called "training-free self-evolution" — no model weights are touched; only project experience accumulates. Games that pass review are distilled into three artifacts.

Skeleton: A runnable template for a game genre, preserving code architecture so new games only need parameter and art changes. Each skeleton comes with an error notebook recording recurring pitfalls in that genre and verified fixes.

Module: A packaged functional script for cross-genre reuse, with self-check logic that converts previously seen runtime crashes into pre-launch static checks — you only need to fall into a pit once.

Contract: A codified workflow for a specific feature type. For example, a map contract specifies how the Designer writes requirements, how the Programmer represents them, how the Artist supplies assets, and how the Auditor and Playtester verify them. When a new project starts, the system retrieves matching skeletons, modules, and contracts and begins from a higher baseline. But experience is never applied blindly — every project re-validates its fit.

At its core, this is the same idea as "feedback-driven learning loops" and "cross-agent unified memory": turn experience into files, and the system grows smarter without retraining.

This "skeleton-module-contract" experience reuse mechanism is conceptually similar to combining RAG (Retrieval-Augmented Generation) with few-shot prompting in software engineering, but deliberately sidesteps fine-tuning. Fine-tuning requires high-quality annotated datasets and compute, with long cycles unfriendly to ordinary developers. Storing experience as structured files and injecting them into context via semantic retrieval at project start requires no changes to model weights. This approach shares roots with external-memory solutions like MemGPT (now Letta): decoupling "memory" from model parameters into a readable, writable, auditable, deletable file system. VibeGame's added emphasis on "re-validating fit for each project" is an important engineering constraint — preventing stale or incorrect experience entries from being applied unconditionally, effectively adding a runtime validation gate to the RAG pipeline.

The Output: From Pixel Sekiro to Role Reversal

The project presents several cases. The text-only Pixel Sekiro starts from a horizontal action skeleton with a single sentence describing the boss fight system, but Sekiro's parry-and-posture mechanic is completely different from a standard health-based combat system — the battle is built from scratch, ultimately delivering 12 scripts, two sets of character assets, 11 particle effect entities, a parry system distinguishing normal blocks from precise parries, and a multi-stage deathblow triggered when posture maxes out.

The multimodal Hollow Knight boss fight uses text to define rules and a concept image to anchor visuals, with the generated ruined chapel tracking the concept image's silhouette, palette, and cool tones.

On the editing side, three levels of capability are demonstrated. The shallowest is IP transfer: swap the entire boss fight to a Kung Fu Panda theme, regenerate all art, leave combat logic untouched — validating the engine's code-asset decoupling. The middle level is genre transfer: convert a turn-based card game to real-time action, completely replacing the gameplay while preserving the cyberpunk visual style, with the system itself judging what to keep and what to rewrite. The deepest is rule transfer: swap the player and boss roles in the Hollow Knight fight — the Knight becomes an AI-driven boss, the antlered boss becomes player-controlled, with input rebinding and win/loss conditions remapped, all without adding new art.

The project page also includes 9 demos — a civilization prototype, arcade fighter, dungeon crawler, Fruit Ninja, endless runner, AI-driven Werewolf, and more — all playable directly in the browser.

Three Things Worth Borrowing, Three Buckets of Cold Water

First, validate through interfaces rather than intuition. Turn "is the game good?" from a human subjective judgment into an agent-executable action sequence: pause, step, inject input, screenshot, generate indicator images. If it can be verified, it can be delivered. Every team building generative systems should learn this.

Second, separate implementation from evaluation. The value of multi-agent systems isn't in having more agents — it's in adversarial roles. Producers shouldn't grade their own work. This principle is worth more than any prompt engineering trick.

Third, the experience triad. Skeletons, modules, and contracts form a training-free self-evolution loop that any developer can replicate in their own workflow today.

But the cold water is real too. The paper itself admits that all capabilities are supported by qualitative descriptions only — no controlled experiments, no quantitative benchmarks; those are left for future work. The system only supports 2D web games; whether 3D spatial physics and cameras can maintain an all-text representation is something the authors say they don't know. Validation itself isn't perfect either — visual judgment can be wrong, and self-evolution carries contamination risk: one bad experience entry in the library can skew future projects, which is why the user confirmation step cannot be skipped.

The project currently sits at 226 stars with two contributors, has no official release, and requires your own Claude Code or Codex plus an image generation API key to run — the long-term barrier is real. It's quite impressive as research; it's too early as a production tool. The project is open-sourced under the Apache 2.0 license.

One final thought: the barrier to making games has never just been writing code — it's also art, tuning, testing, and a whole set of craft skills. What VibeGame suggests isn't that those skills disappear, but that they've been broken apart and distributed to a tireless team. Your job becomes articulating your vision clearly. The protagonist of the next indie game might genuinely be someone who knows how to tell a good story.