Claude Plays Games on Its Own? Open-Source MCP Lets AI Play GTA in the Background

A developer shared a new project on Reddit: an MCP server built with Claude Code that lets Claude autonomously play PC games in the background. From driving in GTA IV to helicopter takeoffs and landings in GTA V, and even making story choices in Detroit: Become Human, everything in the videos is real-time and unedited. What's even more interesting is that Claude wrote most of the code for this project itself.

What Is the Project: Giving AI Its Own Input/Output Devices

According to the original Reddit post, the core idea of this MCP (Model Context Protocol) server is to allocate Claude a dedicated set of keyboard, mouse, gamepad, and screen capture channels that only affect a single game window. The game runs in the background, allowing the developer to use the computer as usual without interference. Multiple games can run simultaneously, each with its own independent Claude agent.

The technical implementation is split into two layers: one is a Windows DLL running inside the game process, responsible for injecting input and performing screen capture; the other is a TypeScript MCP server built on top, exposing approximately 100 tools. The project is open-sourced under the MIT license and is compatible with any MCP client. Currently, it only supports Windows because the in-game component is a Windows DLL.

MCP (Model Context Protocol) is an open protocol introduced by Anthropic in late 2024 to standardize how AI models connect with external tools and data sources. Think of MCP as the "USB-C interface" for the AI world: the server side exposes a set of capabilities (reading/writing files, calling APIs, controlling software, etc.) as named "tools" with parameter descriptions, and the client (such as Claude Code, Claude Desktop, or other compatible applications) initiates requests when the model decides to call a tool and returns the results to the model. MCP uses JSON-RPC transport and supports both local process (stdio) and remote (HTTP/SSE) connections. It contains no game logic itself; rather, it enables Claude to "see" which tools are available and combine them as needed. The approximately 100 tools mentioned in the article are a set of callable actions registered with the model via the MCP protocol.

DLL injection is a common technique on the Windows platform: loading a dynamic link library (DLL) into the address space of a running process to execute code within that process. In this project, the in-game component needs to send input events (keyboard, mouse, gamepad) and capture rendered frames directly within the game process. Ordinary external programs can only interact with the foreground window or the entire screen and cannot precisely target a single background window, hence the injection approach. The injected DLL can call input interfaces like DirectInput and XInput, as well as graphics API capture interfaces to obtain frame data, then pass the results to the external TypeScript server via inter-process communication. Note that this type of technique is also the implementation path for many game cheats, so it only has legitimate use in single-player offline environments without anti-cheat systems.

Real-World Performance: Driving, Flying, and Walking All Work

The demo videos cover multiple game genres. In GTA IV, Claude can drive to map markers and walk to specified locations; in GTA V, it flies a helicopter to a marker and lands. It also made choices in a scene from Detroit: Become Human and tried TABS, Halo Wars 2, Phantom Rose, and Cities: Skylines.

These tasks share a common trait: clear objectives and visual navigation cues. The developer admits that the truly difficult scenarios are indoor environments without minimaps—in The Elder Scrolls III: Morrowind, Claude took considerable time to find the correct door. Aiming in shooter games is also still in its early stages. Full game completion is the goal but has not yet been achieved.

The "visual navigation cues" mentioned in the article refer to guidance information in the game screen that can be directly recognized by vision models, such as target markers on the minimap, waypoint arrows on roads, and directional indicators on screen. The value of these cues lies in decomposing long-horizon tasks into short feedback loops of "current frame → relative direction to target → adjust input," so the model doesn't need to build a complete environment map—it only needs to make a local judgment every frame or every few frames. Conversely, indoor scenes without minimaps lack such external anchors, and the model can only rely on continuous memory of scene geometry and exploration strategies, with errors accumulating over steps—this is precisely why finding the door in The Elder Scrolls III: Morrowind was difficult. The challenge of shooter aiming is different: it requires sub-frame-level continuous fine-tuning and target tracking, making it far more sensitive to input frequency and visual feedback latency than driving-type tasks.

The Development Approach Is the Real Highlight: Agents Write Code and Test Themselves

The most intriguing part of this project isn't "AI can play games" but "AI improves itself in order to play games." According to the original post, Claude Code was responsible for setting up the project, creating plans, and writing most of the code, and the development work itself was done by Claude agents.

Each agent is assigned a game as a testbed along with a specific problem, such as "this car always stalls at this intersection." It actually plays through the MCP, identifies where the tool fails, modifies the code, and measures performance before and after the change on the same route. Claude Code then reviews the changes and runs tests, and only after passing are they made public.

Game adaptation is abstracted into lightweight "configuration files": where the minimap is located, how the camera rotates, what waypoints look like. Supporting a new game is therefore more about adding data than writing code, and most of these configuration files are written by Claude agents themselves during gameplay. The developer mentions that Codex also contributed some of the code work.

Boundaries and Limitations: Single-Player, Offline, No Anti-Cheat

The project currently has clear applicable boundaries. It only supports the Windows platform; it only works with single-player and offline games and cannot run in games with anti-cheat mechanisms; indoor scenes without minimaps, shooter aiming, and full game completion remain unsolved problems.

From a broader perspective, the value of such tools isn't about letting AI "entertain itself" but about providing an automated verification path for game mod development—agents can actually run games, observe behavior, identify defects, and iterate on fixes, forming a closed loop. For mod developers who need extensive repetitive testing, this "let AI play for itself to watch" model may be more practical than pure code generation.

The project is open-sourced on GitHub, and the developer has invited the community in the original post to suggest which game they'd like to see it try next.

One-line Summary

A developer built an MCP server with Claude Code that lets Claude autonomously play PC games in the background and improve its own code.

Paragraph Summary

A developer has open-sourced an MCP protocol-based server that allocates Claude dedicated keyboard, mouse, gamepad, and screen capture channels, enabling it to autonomously play PC games within a single background window without interfering with the developer's normal computer use. Demos include driving and walking in GTA IV, helicopter takeoffs and landings in GTA V, story choices in Detroit: Become Human, and more. Technically, a Windows DLL inside the game process handles input injection and screen capture, while a TypeScript MCP server on top exposes approximately 100 tools. More notably, the development approach: Claude agents use each game as a testbed, identifying tool defects through actual gameplay, modifying code, and comparing performance before and after on the same route, with game adaptation abstracted into configuration files. The project is MIT open-source, compatible with any MCP client, and only supports Windows single-player offline games.