Clash of Clans Mobile Game Screen Segmentation Dataset: 6.9k Frames, 1.9k Classes

New Fuel for Mobile Game Vision AI: A Segmentation Dataset Built Specifically for Clash of Clans

In the field of mobile game AI, high-quality annotated data has always been a scarce resource. Recently, the Reddit community shared an instance segmentation dataset built specifically for Clash of Clans gameplay screens, containing 6,909 frames and approximately 4.2 million masks, covering 1,900 classes. This dataset attempts to precisely outline every visible object on the game screen—from buildings and walls to troops, heroes, and even the army bar and HUD text—using polygons.

For developers looking to train game understanding models, this undoubtedly provides a ready-made starting point. But its value and limitations are equally obvious.

What the Dataset Covers

According to the publisher's description, the dataset frames come from three key Clash of Clans interfaces: the scout screen, the battle, and the end of battle screen. The annotation method is instance segmentation, where each object is represented by a polygon formed by its visible pixels, rather than a simple bounding box.

In terms of data scale, 6,909 frames correspond to approximately 4.2 million masks, with the number of classes reaching 1,900. Both YOLO seg and COCO formats are provided, making it easy to plug into different training pipelines. The validation set is split by base, meaning frames from the same base will not appear in both the training and validation sets, which helps evaluate the model's generalization ability to new bases.

The publisher provides a free sample and an online viewer (Hugging Face Spaces), while the full dataset requires payment.

Instance segmentation differs from semantic segmentation and object detection in key ways: object detection only provides bounding boxes; semantic segmentation assigns a class label to each pixel but does not distinguish between individuals of the same class; instance segmentation requires distinguishing both class and individual, meaning each object (such as two archer towers of the same level) has its own independent mask. The dataset in this article represents masks as polygons, which are more compact than pixel-level masks and easier to store and convert, but they also require annotators to precisely outline object contours. The YOLO seg format is an instance segmentation annotation format supported by the Ultralytics framework, typically storing polygon coordinates in text files; the COCO format is a more general JSON structure widely supported by Detectron2, MMDetection, and others. The coexistence of both reduces the migration cost of integrating the dataset into different training pipelines.

Why Splitting the Validation Set by Base Matters

In game screen segmentation tasks, if frames are split randomly, consecutive frames from the same base may end up in both the training and validation sets, causing the model to achieve artificially high validation scores by memorizing background layouts. Splitting the validation set by base forces the model to learn the visual features of the objects themselves rather than memorizing the arrangement of a specific base. This design reflects the publisher's consideration of the data leakage problem and makes the validation results more meaningful.

For researchers studying game scene understanding, this splitting method is worth emulating.

Data leakage refers to training data containing information highly similar to or even identical to validation or test data, resulting in inflated model evaluation scores that do not reflect true generalization ability. In video or consecutive frame scenarios, the background layout of adjacent frames is almost identical; if split randomly, the model may "pass" validation by memorizing the background rather than learning object features. Splitting by base is a group split strategy that ensures all frames from the same base appear in only one split. Similar practices have been widely adopted in medical imaging (splitting by patient) and autonomous driving (splitting by scene/road segment). When evaluating the effect of group splitting, one typically compares the gap in validation metrics between random splitting and group splitting; the larger the gap, the more severe the leakage.

1,900 Classes: Fine-Grained Detail and Challenges Coexist

1,900 classes is a fairly large number. In Clash of Clans, the same building at different levels looks noticeably different, and troops and heroes have multiple forms. Fine-grained annotation enables the dataset to support detailed parsing of game states, such as identifying defense buildings of a specific level or the number of a specific troop type.

But a large number of classes also brings challenges: the class distribution may be highly imbalanced, with some rare classes having very few samples, making them difficult for the model to learn well. The publisher did not provide class distribution statistics in the description, which is something users need to explore on their own.

Long-tail distribution refers to a highly imbalanced number of samples per class, where a few head classes account for most samples and a large number of tail classes have very few samples. In a segmentation task with 1,900 classes, low-level buildings, rare troops, or specific HUD elements may have only dozens or even a few samples. Long-tail distribution can significantly degrade segmentation accuracy on tail classes. Common strategies include: resampling (oversampling tail classes), loss function weighting (such as focal loss), classifier rebalancing (such as logit adjustment), and merging tail classes into superclasses. Users should first compute a class frequency histogram to identify the degree of long-tailedness, then decide whether to adopt these strategies.

Payment Model and Community Response

The dataset uses a free sample plus paid full version model. This model is not common in the academic dataset space, but it has its rationale in niche vertical domains—annotation costs are high, and making it completely free is difficult to sustain. However, this may also limit its adoption in academic research. The publisher placed the sample and viewer on Hugging Face Spaces, lowering the barrier for potential users to evaluate it, which is a pragmatic approach.

Potential Applications and Limitations

The most direct application of this dataset is training game screen segmentation models, which can then support scenarios such as game state recognition, automated operation assistance, and live stream screen analysis. For strategy games like Clash of Clans, accurately identifying building layouts and troop positions is the foundation for building AI players.

The limitations are also obvious: the dataset covers only one game, and transferring to other games requires re-annotation; the long-tail distribution of 1,900 classes may affect model performance; the paid access model may hinder academic reproducibility. In addition, the publisher did not disclose annotation quality evaluation metrics (such as annotation consistency), so users need to verify it themselves.

Annotation consistency is a core metric for measuring annotation quality, typically evaluated by having multiple people annotate the same batch of images and then computing Intersection over Union (IoU) or the Dice coefficient. For instance segmentation, it is also necessary to examine differences in how different annotators determine the boundaries of the same object, especially in occluded areas, semi-transparent effects, and overlapping HUD regions. If the publisher does not disclose consistency metrics, users can indirectly judge the level of annotation noise from the loss gap between training and validation sets and from visualized model results on samples. When annotation noise is too high, the model may learn incorrect boundaries; in such cases, annotation smoothing or noise-robust loss functions can be considered.

Conclusion

This Clash of Clans segmentation dataset fills a gap in the field of mobile game screen understanding. With its scale of 6.9k frames, 4.2 million masks, and 1.9k classes, plus the rigorous design of splitting the validation set by base, it is a resource worth the attention of game AI researchers and developers. Whether to pay for access depends on the specific project's requirements for data scale and quality. For academic research, the free sample can serve as a good starting point.