17 Million Mouse Trajectories: The Real ML Value of Gaming Behavioral Data

One Gamer's Accidental Data Treasure

Recently, a League of Legends player shared a personal side project on Reddit: by recording in-game mouse telemetry, clicks, and keyboard inputs synchronized with video footage, he accumulated 17 million mouse cursor position records and 670,000 click events across approximately 350 matches.

The poster openly admitted he isn't an ML professional, and asked the community: does this kind of gaming behavioral data actually have practical ML applications? This seemingly simple question touches on several cutting-edge fields — including game behavior modeling, player input analysis, and reinforcement learning.

Data Collection and Structure

Based on the schema the poster shared publicly, the collection system is remarkably complete. Each match is recorded in structured JSON format across the following dimensions:

• Game metadata: game mode (e.g., ranked solo/duo), champion played (e.g., Varus), win/loss outcome, KDA, match duration, screen resolution, sampling rate (63Hz)
• positions: timestamped mouse coordinate sequences (t, x, y)
• clicks: click events including time, coordinates, and button type (left/right)
• keypresses: key records including key type and duration
• gameEvents: key in-game events (e.g., deaths with timestamps)
• abilitySnapshots: ability state snapshots (time-series of Q/W/E/R and HP values)

A 63Hz sampling rate means 63 cursor positions are recorded per second — precise enough to capture fine-grained input behavior. For context, modern gaming peripherals typically run polling rates between 125Hz and 8000Hz, but 63Hz already covers most consciously controlled human movements — the upper frequency limit of fine motor control is roughly 10–15Hz, so 63Hz oversampling provides ample temporal resolution.

The more technically challenging aspect is multimodal time synchronization: precisely aligning OS-level mouse events (captured via Raw Input API or similar mechanisms) with in-game event timestamps and video frames requires handling clock drift across multiple data streams. Commercial precedents for this kind of collection system exist in game testing and esports analytics platforms like Overwolf's overlay tools, but open-source implementations that align five data streams to frame-level precision remain rare. Most critically, this dataset achieves multimodal temporal synchronization across mouse, keyboard, ability states, game events, and video — and that is precisely what sets it apart from ordinary log data.

What Can Gaming Behavioral Data Be Used For?

Player Behavior Modeling and Input Analysis

The most immediate application is player input behavior modeling. 17 million time-ordered mouse trajectories, paired with click and ability cast data, can reconstruct a player's decision-making patterns across different in-game contexts. For example:

• Analyzing mouse micro-adjustment precision during last-hitting (CS)
• Studying APM (actions per minute) fluctuations before and after teamfight engagements
• Building behavioral feature models from the seconds leading up to a death event, identifying input patterns correlated with mistakes

Since the data includes both win/loss outcomes and KDA, researchers could even extract signals correlated with game performance from input features — potentially building a practical "input quality score" tool.

Training Material for Imitation Learning and Reinforcement Learning

For game AI research, this type of human demonstration data holds special value. Imitation Learning is an important subfield of reinforcement learning, centered on the idea of having an agent extract behavioral policies from expert demonstrations — bypassing the challenge of manually designing reward functions. Key approaches include Behavioral Cloning (BC) and Inverse Reinforcement Learning (IRL).

BC simplifies the problem into supervised learning — directly mapping states to actions — but suffers from distribution shift: the state distribution during training often differs from that during execution, causing compounding errors. More sophisticated methods like GAIL (Generative Adversarial Imitation Learning) borrow from GANs, using a discriminator to distinguish expert behavior from agent behavior.

A relevant reference point: DeepMind's AlphaStar extensively used human professional player demonstration data for supervised pretraining in its early stages, before achieving superhuman performance through self-play reinforcement learning. This illustrates that human demonstration data serves more as a "warm start" than a full training signal for game AI. In the imitation learning paradigm, the poster's dataset is essentially a record of state–action pairs: ability snapshots and game events constitute the state, while mouse positions and clicks constitute the actions.

That said, it's worth being objective about the limitations of using this data directly to train a complete League of Legends AI:

• Incomplete state information: The data lacks minimap information, ally/enemy unit positions, gold/economy status, and other critical game state variables
• Single-player perspective: All 350 matches come from one player, making the style highly homogeneous and difficult to generalize from
• Unstructured video: Although video is synchronized, the footage itself hasn't been annotated or parsed

As a result, this dataset is better suited as material for specific input pattern research or as a supplementary dataset, rather than as the primary corpus for training a game AI from scratch.

Human-Computer Interaction and Mouse Behavior Prediction

Beyond gaming itself, pure mouse trajectory data is equally useful in Human-Computer Interaction (HCI). The 670,000 clicks contain rich data relevant to Fitts's Law — a foundational law in HCI proposed by psychologist Paul Fitts in 1954, describing the logarithmic relationship between movement time and target distance and size: MT = a + b·log₂(2D/W), where MT is movement time, D is distance, and W is target width. This law has been widely validated and applied in mouse operation, touchscreen interaction, and menu design.

League of Legends presents a particularly unique environment: large volumes of fast, precise clicks are made within dense clusters of game units. This extreme-condition mouse behavior data has special academic value for validating and extending the boundary conditions of Fitts's Law. Based on large-scale mouse trajectory data, researchers can also build intent prediction models that go beyond Fitts's Law (e.g., LSTM- or Transformer-based trajectory prediction) to anticipate user click targets in advance.

This type of data can be applied to:

• Mouse movement trajectory intent prediction models
• Human behavior verification (distinguishing real players from scripted bots)
• Fatigue detection (analyzing precision degradation after extended gaming sessions)

A Realistic Assessment of Data Value

The Trade-off Between Scale and Quality

17 million data points sounds enormous, but it deserves a measured perspective in the ML context. The value of data depends not just on quantity, but on diversity, annotation quality, and task relevance.

In machine learning, data diversity's impact on model generalization is often more fundamental than raw scale. This was demonstrated repeatedly during the ImageNet era: models that overfit to a specific distribution — so-called "dataset bias" — frequently collapsed in real-world deployment. For behavioral data, the homogeneity problem of single-user data is especially pronounced — one player's input habits, champion preferences, and decision-making style form a unique "behavioral fingerprint," and models can easily learn personal idiosyncrasies rather than universal patterns.

Academic solutions include Domain Randomization, Data Augmentation, and Transfer Learning. The more fundamental fix is a crowdsourcing approach: OpenAI faced similar annotator diversity challenges when training RLHF models and ultimately mitigated them through careful annotator pool management — a logic directly parallel to building a multi-player dataset.

This dataset's greatest strength is its multimodal synchronization — five streams of data (mouse, keyboard, ability states, game events, video) aligned together, a level of careful collection rarely seen in public datasets. Its greatest weakness is its single source: everything comes from one player, lacking population-level diversity, which directly limits model generalizability.

More Realistic Use Cases

Overall, the most likely high-value scenarios for this gaming behavioral dataset are:

1. Personal skill diagnostic tools: Quantifying a player's own input weaknesses and improvement trajectory
2. Small-scale behavioral research: As a behavioral analysis sample for academic or hobbyist projects
3. Demonstration of a telemetry collection system: The accompanying recording infrastructure is itself a valuable open-source engineering project

The poster has already open-sourced a complete sample dataset on GitHub, along with a visualization demo of gameplay replays. This spirit of open sharing is exactly the right path to amplifying the value of a niche dataset — if more players adopted the same tool to collect data, a multi-player, multi-champion crowdsourced dataset would emerge, and its ML value would increase by orders of magnitude.

Conclusion

This case reveals an interesting reality: ordinary users can produce high-quality structured behavioral data through their everyday activities. This data may not directly train a spectacular AI model, but it is raw ore for behavioral science, human-computer interaction, and game intelligence research.

To the poster's question — "Is this data useful?" — the answer is yes. The key is clarifying the task objective and expanding data diversity. A high-quality telemetry dataset from a single player may not sustain a complete game AI, but it can absolutely serve as a precise window into understanding human gaming behavior.