Open-Source Tool: Automatically Collect 800K High-Elo LoL Match Data Using the Riot API

From Gaming Passion to Data Science Project

For developers who love both League of Legends (LoL) and data science, a natural question arises: can a machine learning model (like XGBoost) predict match outcomes purely based on team compositions and champion combinations?

Background on XGBoost: XGBoost (eXtreme Gradient Boosting) is an ensemble learning algorithm based on gradient-boosted decision trees, proposed by Tianqi Chen in 2014. It works by sequentially building multiple decision trees, where each tree attempts to correct the prediction errors of the preceding trees, ultimately combining all trees' predictions through weighted aggregation. XGBoost has excelled in data competitions like Kaggle and is particularly adept at handling tabular data and structured data with feature engineering. Compared to traditional gradient boosting algorithms, XGBoost introduces regularization terms to prevent overfitting, supports parallel computation for faster training, and has built-in missing value handling. For classification tasks like predicting game outcomes, XGBoost can automatically capture nonlinear interactions between champion combinations without requiring manually designed complex feature engineering.

This is exactly what motivated the open-source project lol-dataset-generator. As shared by a developer on Reddit, they hit a core pain point while building a prediction model — the lack of a clean, large-scale, publicly available dataset for the current season. Existing LoL datasets on the market were either outdated or poorly formatted, making them difficult to use directly for machine learning training.

To solve this problem, the developer built a complete dataset generator from scratch using Python and the official Riot API, and fully open-sourced it.

The Riot Games API Ecosystem: Riot Games provides developers with a comprehensive set of RESTful APIs that allow third-party applications to access League of Legends game data. The API is divided into multiple endpoints, including summoner information, match history, leaderboards, champion mastery, and more. Developers need to register on the Riot Developer Portal and apply for an API key — free developer keys come with strict rate limits (typically 20 requests per second and 100 requests per 2 minutes). The API returns JSON-formatted data containing detailed match timelines, kill events, economy data, and more. Notably, the Riot API uses a region-based architecture, meaning different servers (e.g., NA, EUW, KR) require corresponding regional endpoints. The API also enforces data retention policies, typically only allowing access to match data from the past few months — one of the reasons continuous data collection is necessary.

The project is hosted on GitHub, and anyone can run it with just a free Riot developer API key.

Core Functionality: Automated High-Elo Data Collection

Focusing on High-Quality Match Samples

The tool's core capability lies in automatically targeting high-quality matches. It automatically fetches Challenger, Grandmaster, and Master tier players across all regions. This design choice is deliberate — high-elo matches better reflect patch-specific champion strength and authentic strategic gameplay, with less noise and higher data quality for training prediction models.

Statistical Significance of High-Elo Data: In machine learning, sample quality often matters more than quantity. League of Legends' ranking system follows a pyramid distribution, with Challenger, Grandmaster, and Master tiers comprising only about 0.1% of all players. Yet these high-elo matches have unique value: (1) High decision quality: High-elo players have deep champion understanding, and their pick/ban choices are closer to the theoretical optimum, reducing the "random picking" noise common in lower ranks; (2) Meta-game representativeness: Champion selection trends in professional play and high elo are highly correlated, making this data better at reflecting true patch strength; (3) Consistent mechanical execution: High-elo players can fully leverage a champion's potential, so match outcomes depend more on composition and strategy than mechanical errors. From a statistical perspective, this is essentially a "quality filter" on the sample space — sacrificing randomness but improving the signal-to-noise ratio. However, this also introduces limitations — the model may perform poorly for lower-elo predictions, where different playstyles and champion ecosystems exist.

The collection process has three steps:

1. Retrieve the list of high-elo players from each region
2. Collect Ranked Solo/Duo match IDs from these players
3. Extract match details and flatten them into clean, ML-ready CSV files

Structured Feature Fields

The generated dataset includes key features valuable for modeling:

• Patch version: LoL releases balance adjustments roughly every two weeks, and the same champion's strength can vary dramatically across patches — this field is critical for model generalization

Impact of Game Patches on Data Validity: League of Legends employs a rapid-iteration balancing mechanism, releasing a new patch approximately every two weeks. Each patch may adjust champion ability damage, cooldowns, base stats, or change items and rune systems. This means the same champion's power level can shift drastically between patches — a top-tier champion in one patch might be nerfed into obscurity in the next. Consequently, mixing historical data across patches for training can produce "data drift" problems, where patterns the model learned may no longer apply to the current patch. This is why the dataset must include patch version labels, and modeling should consider patch weighting or use only recent patch data. In professional esports analysis, typically only data from the most recent 2–3 patches is referenced for pick/ban strategy, as older data has significantly diminished relevance.

• Game Duration
• Champion roles for both teams: Top, Jungle (Jgl), Mid, Bot, and Support (Sup)
• Average Team Tier
• Match Result (Win/Loss)

This combination of fields covers exactly the information available after the champion select phase ends, providing complete input for building a "predict the winner at draft" model.

CSV Format and ML-Ready Data: An "ML-ready" dataset refers to data that has been cleaned, transformed, and formatted so it can be directly imported into machine learning frameworks for training. CSV (Comma-Separated Values) is the most universal tabular data exchange format, directly readable by mainstream tools like pandas, scikit-learn, and XGBoost. Compared to the nested JSON structures returned by APIs (which may contain arrays, dictionaries, and other complex hierarchies), CSV flattens data into a two-dimensional table where each row represents a match and each column is a feature. This transformation process includes extracting nested fields, encoding categorical variables (e.g., converting champion names to numerical IDs), handling missing values, and unifying data types. For high-dimensional categorical features like champion compositions, One-Hot encoding or entity embeddings are typically used. A well-designed ML-ready dataset eliminates data quality issues, allowing data scientists to focus their efforts on feature engineering and model tuning rather than data cleaning.

Noteworthy Engineering Design Details

SQLite-Based Checkpoint and Resume Mechanism

The aspect of this project that best demonstrates engineering awareness is its built-in SQLite database for tracking collection progress. This means you can interrupt the collection task at any time and resume from where you left off later — without losing existing data or wasting precious Riot API request quota.

SQLite in Data Collection: SQLite is a lightweight embedded relational database that requires no separate server process and stores all data in a single file. In data collection scenarios, SQLite serves as a "state manager," recording metadata such as collected match IDs, player PUUIDs, and collection timestamps. This design achieves idempotency — even if the program terminates unexpectedly, it can query the database upon restart to skip already-collected data and avoid duplicate requests. Compared to simple text logs, SQLite provides SQL query capabilities for quickly checking "which players' matches haven't been collected yet" or "how many samples exist for a given patch." Additionally, SQLite's transaction features ensure data consistency — either all related data for a match is written successfully, or the entire operation is rolled back. For crawlers that need to run for extended periods, this lightweight persistence approach is more reliable than in-memory caching and easier to deploy than heavyweight databases like MySQL.

For those familiar with API development, this design is critical. The Riot API has strict rate limits, and collecting 800,000 matches means millions of API calls that could take days of continuous running. Without a checkpoint-resume mechanism, any network interruption or program crash could render all previous efforts futile. This detail elevates the project from a "script that works" to a "production-grade data collection tool."

Data Scale of 800,000 Matches

The developer's goal is to collect approximately 800,000 matches. This scale is quite substantial for machine learning tasks — sufficient to train models with strong generalization capability while also supporting fine-grained analysis across different patches and champion combinations.

Practical Use Cases for the Dataset

With a large-scale high-elo dataset like this, several valuable analyses and modeling efforts become possible:

• Analyzing win rates during the Ban/Pick phase: Quantifying win/loss tendencies of different team compositions
• Studying champion synergies: Discovering which champion combinations have hidden powerful interactions
• Building draft-phase win prediction models: Predicting which team is more likely to win after champion select ends but before the match officially begins
• Esports pick/ban strategy research: Providing data-driven support for professional teams' composition decisions

This type of research is valuable not only as a reference for regular players looking to climb the ranked ladder, but also has practical significance for esports pick/ban analysis and game balance research.

Takeaways for Developers

Although this project focuses on gaming, it demonstrates a universal paradigm for excellent data engineering projects: a well-defined data source (Riot's official API), a focus on high-quality samples (high-elo matches), a robust collection mechanism (checkpoint and resume), and downstream-task-oriented data formatting (ML-ready CSV).

For developers looking to get started in data science, this is also an excellent hands-on case study — starting from a genuine personal interest and pain point, using engineering methods to solve the "no available dataset" problem, and ultimately producing an open-source tool that serves both oneself and the community.

The developer also actively solicited feedback on Reddit, asking what additional statistical fields should be included in the CSV. Interested readers can visit the GitHub repository mehdbenguiza/lol-dataset-generator to try it out and contribute code.