Game Behavior Data Collection: Frame-Level Aligned Human Action Data for Imitation Learning

The Overlooked Data Gap in Game AI Research

In recent years, research directions such as game agents, imitation learning, and world models have been extremely hot. However, they all face a common bottleneck — a scarcity of high-quality "video-action" paired data. Models need to know exactly which key a human player pressed and how many pixels the mouse moved at any given frame. Without such aligned data, behavioral cloning and policy learning become castles built on sand.

Some background is helpful here: Imitation Learning is an important branch of reinforcement learning whose core idea is to let agents learn policies by observing expert (usually human) behavior, rather than through trial-and-error and reward signals. Behavioral Cloning is the most straightforward approach, essentially converting policy learning into a supervised learning problem — using environmental states as input and expert actions as labels to train a state-to-action mapping function. This method can be traced back to Carnegie Mellon University's ALVINN autonomous driving project in 1989. While behavioral cloning is simple and efficient, it also suffers from the well-known "distribution shift" problem: during training, the agent only sees states near expert trajectories, and once it deviates from those states during actual execution, cumulative errors arise. Subsequent algorithms like DAgger have attempted to mitigate this issue, but high-quality expert demonstration data remains the absolute foundation.

Recently, a developer shared their solution on Reddit: a Windows-based gameplay data collection prototype (gameplay-data-collector). The core value of this tool lies in its attempt to use a "spec-driven" approach to precisely align game visuals with player inputs, producing structured datasets ready for research use. The related code and sanitized samples are open-sourced on GitHub.

What Exactly Does This Collection Tool Record?

According to the author's description, this prototype is designed around a "capture contract" and simultaneously records the following types of data:

Core Capabilities of Multimodal Synchronized Capture

• Target window video: Records visuals from a specified game window rather than full-screen capture, reducing irrelevant information noise.
• Keyboard state (100ms sampling): Records key states at 100-millisecond intervals, capturing sustained presses and key combinations.
• Raw Input mouse data: Records mouse displacement deltas and button events. Using Raw Input instead of system cursor coordinates is especially critical for FPS and action games — because these games typically lock the cursor and rely on relative displacement, and ordinary coordinate capture simply cannot obtain the real operation trajectory.
• Frame-level alignment index: Aligns video frames with input events through frame indices, ensuring that "this frame" corresponds to "this moment's operation."
• Basic QA metadata: Includes quality verification information to facilitate downstream filtering and data reliability validation.

The technical significance of the Raw Input mechanism deserves deeper understanding. Raw Input is a low-level input API provided by the Windows operating system that allows applications to obtain unprocessed raw data directly from input devices. Unlike traditional window messages such as WM_MOUSEMOVE, Raw Input bypasses Windows' cursor acceleration, boundary clipping, and DPI scaling layers. For FPS games, this is crucial: when a game locks the mouse cursor to the window center via DirectInput or Raw Input, the system cursor coordinates remain unchanged, and traditional capture methods can only obtain static coordinate values. Raw Input, however, captures the mouse's raw displacement deltas — which represent the player's actual aiming intent. For example, when a player performs a quick flick shot, Raw Input can record instantaneous displacements of hundreds of pixels, while the system cursor may not have moved at all. This kind of data is irreplaceable for training agents that can simulate human aiming behavior.

The technical complexity behind frame-level alignment also deserves attention. Frame-level alignment may seem simple, but it actually involves synchronization challenges across multiple technical layers. First is timestamp precision: Windows' default timer precision is approximately 15.6 milliseconds (64Hz), while games typically run at 60-144 frames per second (approximately 7-16 milliseconds per frame), meaning high-precision timers (such as QueryPerformanceCounter) must be used to achieve accurate frame-input matching. Second is multi-thread synchronization: video capture, keyboard monitoring, and mouse monitoring typically run on different threads or even different processes, requiring carefully designed synchronization mechanisms to avoid temporal drift. Additionally, there is an inherent delay (capture latency) between when a game frame finishes rendering on the GPU and when the capture tool captures it, and this delay varies significantly across capture methods: DXGI Desktop Duplication typically has 1-2 frames of delay, while the Windows Graphics Capture API may be shorter. These details determine the true precision of "state-action" pairs in the final dataset.

Current Validation Scope

The author acknowledges that this prototype has only been validated on a single Windows machine with The Elder Scrolls V: Skyrim Special Edition. In other words, this is still an early-stage proof of concept, quite far from generalized, large-scale data collection. This is precisely why the author is seeking collaboration with university labs or research teams to verify whether this capture contract can genuinely be embedded into actual research or evaluation workflows.

Why This Type of Game Behavior Data Deserves Attention

Direct Value for Imitation Learning and Behavioral Cloning

The essence of imitation learning and behavioral cloning is letting agents learn policies from human demonstrations. The core of data quality lies in the precision of "state-action" pairs. This prototype's design hits the nail on the head: video represents environmental state, keyboard and mouse trajectories represent human actions, and frame-level alignment ensures strict temporal consistency between the two.

Compared to extracting state from within game engines (which often requires specific APIs or game modifications), this "external black-box capture" approach offers stronger generality — theoretically, any Windows game can be captured without requiring cooperation from game developers.

Significance for World Models and Embodied Intelligence Research

World models attempt to learn the ability to "predict the next frame given the current frame and action." This requires massive sequences of "frame + action + subsequent frame." Similarly, GUI agents and embodied agents research needs to understand "given a certain interface state, what results does a particular operation produce."

The concept of World Models originates from the "mental model" idea in cognitive science, and was systematically proposed in the AI domain by Ha and Schmidhuber in their 2018 paper World Models. Its core architecture typically consists of three components: a visual encoder (compressing high-dimensional pixels into low-dimensional latent representations), a memory module (usually RNN or Transformer, learning environmental dynamic transition patterns), and a controller (making decisions based on latent states). In recent years, world model research has made breakthrough progress: DeepMind's Dreamer series achieved the ability to train policies entirely in "imagination" on Atari and robotics control tasks; in 2024, Google's Genie model learned interactive 2D game world models from internet videos; that same year, the Decart team's GameNGen demonstrated for the first time the ability to use diffusion models to generate playable DOOM visuals in real time. All these advances point to a common requirement: massive amounts of "observation-action-next observation" triplet data.

Games, as closed environments with clear rules, immediate feedback, and low cost, are naturally ideal sandboxes for this type of research. This is why institutions like DeepMind and OpenAI have long used games as testing grounds for agent research. Looking back at this tradition, the role of game environments in AI research can be traced to Shannon and Turing's exploration of chess programs in the 1950s. Modern milestones in game AI research include: DeepMind's DQN first demonstrating the power of deep reinforcement learning on Atari games in 2013; AlphaGo defeating the Go world champion in 2016; OpenAI Five defeating professional teams in Dota 2 in 2019; and DeepMind's AlphaStar reaching Grandmaster level in StarCraft II the same year. Games became ideal research sandboxes because they simultaneously possess several key properties: controllable state space complexity (from simple Atari to complex open worlds), clear reward signals (scores, win/loss), the ability to run massively parallel simulations, and safety (no physical damage or economic loss like real robots).

Video-Action Trajectory Evaluation Benchmarks

Beyond training data, the author also mentions "video-action trajectory evaluation" as an application scenario. After training a game agent, how do we measure whether its behavior is "human-like"? Real human keyboard and mouse trajectory data can serve as a reference benchmark for evaluation.

Practical Challenges and Real-World Considerations

Despite the clear approach, this prototype still faces considerable challenges on the road to practical use:

Insufficient large-scale validation: Single-machine, single-game validation is insufficient to prove robustness. Different games' window mechanisms, rendering methods, and anti-cheat systems may all create capture obstacles. In particular, many online games' anti-cheat modules may misidentify such input monitoring tools as cheating software.

Understanding the complexity of modern game anti-cheat systems is important here. Kernel-level anti-cheat systems represented by EasyAntiCheat, BattlEye, and Riot's Vanguard monitor the behavior of all processes at the operating system kernel level, detecting suspicious activities such as memory read/write injection, API hooks, and input simulation. The working principles of these systems inherently conflict with data collection tools: capture tools need system-level hooks to monitor input events, which is exactly the technical approach commonly used by cheats. Being detected by anti-cheat systems can result in game account bans and even legal risks. This is why the prototype chose The Elder Scrolls V: Skyrim — an offline single-player game — for validation: it has no anti-cheat interference. Extending this approach to online games would likely require establishing formal research partnerships with game publishers.

Data synchronization precision limitations: A 100ms keyboard sampling rate may be sufficient for slow-paced RPGs but could be too coarse for fast-paced fighting or racing games. The accuracy of frame index alignment under high frame rates and frame-drop scenarios also needs further verification.

Privacy and compliance risks: Capturing players' complete operation trajectories involves behavioral data privacy. The author's provision of "sanitized" samples is a good starting point, but authorization and compliance issues during large-scale collection cannot be overlooked.

Copyright boundaries: The legal boundaries of recording commercial game footage for research purposes vary across regions, and research institutions should carefully evaluate before use.

Conclusion

This project initiated by an individual developer, while small in scale, precisely targets a real research pain point: insufficient supply of high-quality, frame-level aligned human game behavior data. Its "spec-driven" design philosophy, along with its attention to details such as Raw Input and multimodal synchronization, demonstrates the author's deep understanding of the research needs in imitation learning and game agents.

For teams working on game agents, imitation learning, or world model research, this type of open-source collection tool could lower the barrier to data acquisition. Of course, it is still in an early stage seeking feedback and collaboration — whether it can truly become a component of research workflows remains to be validated by more labs across a wider range of games and hardware environments.