AI Plays Pokémon Red: Jev Clears Two Gyms for Under $2

A Low-Cost AI Gaming Experiment

A Reddit user shared a fascinating AI experiment: letting an AI model named Jev play the classic game Pokémon Red in real time, while the "harness" (control framework/middleware) was built simultaneously by Opus 5 as the game progressed. This approach — building the toolchain while playing — showcases the flexibility of current AI in autonomous task execution.

According to the poster, Jev has already defeated two gyms in the game, and as of the time of sharing, the total token cost to run Jev was less than $2. That number alone is the most striking highlight of this experiment — it demonstrates that the barrier to having AI play complex games is dropping fast.

Why the Harness Is the Key

Getting an AI to play a turn-based RPG like Pokémon Red isn't as simple as feeding the game screen to a model. The AI needs a middleware layer to read game state, parse visual information, and translate decisions into concrete action commands. This framework is typically called a "harness."

What makes this experiment noteworthy is that the harness wasn't written in advance — it was built in real time by Opus 5. This means the model was continuously refining its tool interfaces with the game while simultaneously advancing through it. This "build-as-you-go" approach tests the AI's engineering capability and task decomposition skills, making the entire process closely resemble a real autonomous agent (agent) workflow.

From a technical implementation perspective, a harness typically needs to handle several core responsibilities: reading game memory state data (character position, HP, inventory items, etc.) via emulator APIs such as PyBoy or BizHawk; converting game screenshots or structured state into text/image inputs the model can understand; receiving action commands from the model and mapping them to actual button inputs; and maintaining a "memory buffer" to prevent the model from forgetting critical progress during long conversations. Opus 5 generating this framework in real time during gameplay is essentially executing a loop of "tool use + self-reflection": the model tries a certain interface design, judges its effectiveness based on game feedback, then iterates and refines. This capability aligns closely with the "LLM as programmer" paradigm that has emerged in recent years, and is an important dimension for evaluating a model's engineering reasoning ability.

Cost Is the Real Signal

In the past, having large models complete long-horizon, multi-step gaming tasks often meant steep inference costs. The record of clearing two gyms for under $2 reflects two trends:

• Improved model efficiency: The latest generation of models consume fewer tokens to accomplish the same tasks.
• More precise task orchestration: Well-designed harnesses reduce unnecessary inference calls and avoid wasting tokens.

For developers and enthusiasts, low cost means these kinds of experiments are no longer exclusive to well-funded labs. Anyone can run their own AI gaming challenge on a budget of just a few dollars.

The Value of Using Games to Test AI Capability

Pokémon Red as a vehicle for AI benchmarking is no accident. Games like this incorporate long-term memory, map navigation, battle strategy, resource management, and other multi-layered challenges, making them ideal for comprehensively testing a model's planning and execution abilities. Unlike one-off Q&A tasks, completing a game requires the AI to maintain goal consistency across hundreds or even thousands of steps.

Anthropic has previously used the Pokémon series as a reference for measuring Claude's long-horizon capabilities. This community-driven experiment, in a sense, continues that line of thinking — using a game familiar to everyone to intuitively demonstrate the boundaries of AI capability.

Pokémon Red was released in 1996 and is a classic turn-based RPG on the Game Boy platform. The full game includes 8 gyms, 151 catchable Pokémon, and a linear main storyline. From an AI benchmarking perspective, its advantages are: a bounded state space but extremely deep decision tree (completing the game requires hundreds of effective operations); game rules that are highly transparent to human players, making it easier to evaluate whether the AI "truly understands" rather than randomly guessing; and a mature community toolchain that enables low-cost integration. In 2024, a project called "Twitch Plays Pokémon AI" and internal Claude evaluations at Anthropic both used it as a test scenario. Compared to abstract math reasoning problems or code generation tasks, game completion progress is an intuitive and quantifiable milestone, making capability comparisons across different models and time points much clearer.

Closing Thoughts: A Small Experiment with Big Implications

This Reddit post isn't heavy on details, but the direction it reflects is worth paying attention to: AI agents are moving from "answering questions" to "completing tasks," and the cost of completing those tasks is falling rapidly. When an AI can play a classic game for under $2 while building its own toolchain from scratch, there's good reason to be optimistic about more complex and practical autonomous agent applications.

Interested readers can follow the original poster's updates to see how far this AI Pokémon challenge can go.