Reddit Buzz: Building a Superhuman Slay the Spire 2 AI Tutor with Reinforcement Learning

In the field of game AI, games like chess, Go, and Dota 2 have been conquered by neural networks, but deck-building roguelikes have always been a tough nut to crack. Recently, a Reddit post sparked discussion among reinforcement learning (RL) enthusiasts: a user has launched the OstyAI project, aiming to build a "superhuman" AI tutor for Slay the Spire 2 (STS2) to help players understand this game that is extremely unfriendly to neural networks.

Why Is Slay the Spire 2 So Difficult for Neural Networks?

Unlike perfect-information games like chess, deck-building games are full of imperfect information and randomness. Players cannot predict the enemies in the next room, the cards or relics that will drop, which requires the AI to have long-term planning ability under uncertainty.

Moreover, the action space of STS2 is enormous: each turn, players need to choose which card to play from their hand, which potion to use, and even decide whether to end the turn. As the deck and relics grow, the state space expands exponentially. The post author points out that this game seems "particularly difficult for neural networks to learn to a level comparable to humans," which is exactly where the research value lies—it may push the boundaries of RL in complex decision-making scenarios.

Imperfect-information games are a core category in game theory and reinforcement learning. Unlike "perfect-information" games such as Go and chess, where both sides can see the entire board, in imperfect-information games players can only observe part of the state, such as the opponent's hand in card games or the unknown cards on top of the deck. Classic solutions include incorporating belief distributions into the state representation (e.g., using "the probability distribution of unknown cards" as input features), or using specialized algorithms such as counterfactual regret minimization (CFR). When deep reinforcement learning handles such problems, it usually requires recurrent neural networks or Transformers to maintain a belief state over hidden information, which significantly increases modeling difficulty.

Core Ideas of the OstyAI Project

The initiator positions OstyAI as a "superhuman STS2 teacher," meaning the AI should not only be able to complete runs but also explain its decisions to help human players improve. This "tutor-style AI" approach contrasts with the traditional goal of "beating humans" and places more emphasis on interpretability and educational value.

The post calls on all enthusiasts interested in RL, playing STS2, or researching high-difficulty game AI to join. Although project details are currently limited, judging from the title and content, the initiator hopes to leverage community power to jointly explore how to apply RL techniques (such as deep Q-networks, policy gradient methods, or Monte Carlo tree search) to this game.

Monte Carlo tree search (MCTS) is a decision-planning algorithm based on random simulations, which shone brightly when combined with neural networks in AlphaGo. Its basic idea is to start from the current state and build a search tree through repeated "selection—expansion—simulation—backpropagation," using statistical win rates to guide action selection. In games like Slay the Spire 2, MCTS faces two major challenges: first, random events (card draws, enemy intents) make the branching factor of the tree extremely large; second, a single simulation requires fully simulating an entire combat, which is computationally expensive. Therefore, practical applications often need to combine policy networks for prior guidance or use determinization techniques to approximate randomness by sampling.

Community Reaction and Research Significance

According to the discussion atmosphere under the Reddit post, many players expressed interest in the "AI tutor" concept. Some comments argued that if the AI can explain why a certain card is the optimal play in the current situation, it would greatly help newcomers understand the game mechanics. Researchers also pointed out that the randomness of STS2 makes the reward signal sparse, and training difficulty is far higher than in deterministic games; imitation learning or curriculum learning may be needed to accelerate convergence.

From a broader perspective, game AI has always been a testing ground for general artificial intelligence. The complex decision space of Slay the Spire 2 may become a new benchmark for testing the robustness of RL algorithms. If the OstyAI project makes progress, its methods may be transferable to real-world imperfect-information decision-making scenarios, such as financial investment or medical diagnosis.

Sparse rewards are a typical challenge in reinforcement learning: the agent receives zero reward at most time steps and only gets a signal when the game is won or lost, making credit assignment extremely difficult. In Slay the Spire 2, a single run may last hundreds of decision steps, while the win/loss signal appears only at the end. Common mitigation methods include: designing dense rewards (e.g., giving intermediate rewards based on floor count or health changes), using imitation learning to pretrain policies from human gameplay, adopting curriculum learning to gradually increase difficulty, and using hindsight experience replay (HER) to relabel effective segments in failed trajectories. These methods are often combined in sparse-reward scenarios.

How to Participate or Follow

For developers who want to join, it is recommended to start by familiarizing yourself with the game mechanics and refer to existing open-source implementations of Slay the Spire AI (e.g., building the environment with Python and PyTorch). The post itself is a call to action; participants can proactively contact the initiator and contribute game environment wrappers, algorithm design, or evaluation metrics.

Although OstyAI is still in its early stages, the question it raises—how to make neural networks reach human level in deck-building games—is worth pondering for every RL enthusiast.