Pixel-Art MDP Simulator: An Interactive Way to Learn Core Reinforcement Learning Algorithms

When learning reinforcement learning, the Markov Decision Process (MDP) is often the most abstract and difficult first hurdle. Concepts like the Bellman equation, value iteration, and policy iteration are easy to grasp when you look at the formulas, but truly understanding them requires hands-on practice. Developer xavendano recently released an interactive pixel-art simulator called Markov Kingdom on the Reddit community, aiming to lower the barrier to entry for MDP and reinforcement learning through gamification.

A Completely Free Pixel-Art Learning Tool

Markov Kingdom has a clear positioning: letting learners intuitively experience how MDPs work in a visual environment. It is completely free, publicly hosted on Vercel, and the source code has been released on GitHub. According to the developer's introduction on Reddit, the project's original intention was to "experiment with and better understand reinforcement learning, especially Markov Decision Processes."

It supports both Spanish and English, which is a practical reduction of barriers for non-native English speakers. The pixel-art aesthetic makes the entire learning process feel more like playing a retro game rather than facing cold mathematical formulas.

Why Is MDP Hard to Learn?

The Markov Decision Process is the mathematical foundation of reinforcement learning. It describes the process by which an agent takes actions in an environment, receives rewards, and transitions to new states. The core difficulties lie in: the definitions of state value functions and action value functions, understanding the Bellman optimality equation, and how to approach the optimal policy through iteration.

Traditional teaching methods usually start with formula derivations. Learners can calculate answers on paper but struggle to build intuition. Markov Kingdom's approach is the reverse—first letting users see changes in states and rewards in an interactive interface, then gradually guiding them to the algorithmic level. This "intuition first, formalism later" path is friendlier for many beginners.

The formal definition of a Markov Decision Process (MDP) consists of a five-tuple: a state set S, an action set A, transition probabilities P(s'|s,a), a reward function R(s,a), and a discount factor γ. The "Markov property" means that, given the current state and action, the probability distribution of future states is independent of earlier history. It is precisely this property that allows MDPs to be solved recursively, and it is also the premise on which the Bellman equation holds. The discount factor γ takes values between 0 and 1 and determines how much the agent values future rewards: the closer γ is to 1, the more patient the agent; the closer γ is to 0, the more short-sighted. Understanding the role of each of these five elements is the starting point for understanding all subsequent reinforcement learning algorithms.

Core Algorithms Covered: From Bellman to SARSA

According to the project description, Markov Kingdom covers several key algorithms in the introductory stage of reinforcement learning:

Bellman Equation and Value Iteration

Value iteration is one of the fundamental methods for solving the optimal policy of an MDP. By repeatedly updating the value estimate of each state, it eventually converges to the optimal value function. In the simulator, learners can observe how values gradually propagate and stabilize through iterations.

The state value function V(s) represents the expected discounted cumulative reward an agent can obtain starting from state s and acting according to a certain policy; the action value function Q(s,a) is the expected return obtained by first executing action a in state s and then acting according to the policy. The two can be converted into each other: V(s) is the weighted sum of Q(s,a) by policy probabilities, while Q(s,a) equals the immediate reward plus the discounted expected value of the next state. The Bellman equation writes this recursive relationship as an equation, allowing values to be computed iteratively in a "bootstrapping" manner. Understanding the correspondence between V and Q is key to distinguishing subsequent methods such as value iteration, policy iteration, and Q-learning.

Policy Iteration

Policy iteration alternates between policy evaluation and policy improvement until the policy no longer changes. Its difference from value iteration is a classic interview question in reinforcement learning, and an interactive simulator can make this distinction concrete and visible.

Policy evaluation means that, with the current policy fixed, repeatedly iterating the Bellman expectation equation until the state value function converges, yielding the true value of that policy. Policy improvement means selecting, for each state, the action that yields higher expected return based on the state value function or action value function obtained from evaluation, thereby producing a new policy that is no worse than the original. Policy iteration alternates these two steps, and each policy improvement guarantees that the policy becomes monotonically better; therefore, in finite MDPs it necessarily converges to the optimal policy within a finite number of steps. Its difference from value iteration is that value iteration compresses evaluation and improvement into a single update, directly iterating on the optimal value function, which is usually cheaper per round but requires more rounds.

Q-learning and SARSA

These two are representative algorithms for temporal-difference (TD) control. Q-learning is off-policy, while SARSA is on-policy—this distinction is often glossed over in a sentence or two in textbooks, but in the simulator, learners can switch algorithms, adjust parameters, and observe the actual differences in policy.

Temporal Difference (TD) methods lie between Monte Carlo and dynamic programming. They do not require a complete model of the environment, nor do they need to wait until the end of an episode to update values. Q-learning's update target is the immediate reward plus the discounted maximum Q value of the next state, so regardless of the behavior policy the agent actually follows, it is approximating the optimal action value function, making it an off-policy method. SARSA's update target instead uses the Q value of the next action actually taken by the agent, so its learning result depends on the current behavior policy, making it an on-policy method. In environments with strong exploration and high risk, the two may converge to different policies: Q-learning is more aggressive, while SARSA is more conservative.

Who Is It For?

Markov Kingdom's target users are fairly clear: students or career switchers who are new to reinforcement learning and are currently struggling through the MDP portion. If you can already skillfully derive the Bellman equation, the value of this tool may be limited. But if after reading the textbook you still feel that "you recognize every word but don't know what it means when put together," an interactive simulator offers a different path to understanding.

The developer is also openly soliciting feedback on Reddit. The project is in an early stage, and its features may continue to iterate based on community input.

Open Source and Extensibility

The project source code is hosted on GitHub (xavendano/markov-kingdom) and released as open source. This means learners with some programming background can read the implementation code, or even extend it with new algorithms or environments. For those who want to move from "using a tool" to "understanding how the tool is implemented," this is a good starting point.

The project is currently hosted on Vercel and can be accessed for free. In the post, the developer welcomed any comments and feedback.

One-line Summary

Markov Kingdom, a free pixel-art interactive simulator, lets beginners intuitively understand MDPs and core reinforcement learning algorithms through gamification.

Paragraph Summary

Markov Kingdom is a free, open-source pixel-art interactive MDP simulator released by developer xavendano on Reddit, designed to lower the barrier to entry for reinforcement learning. It supports English and Spanish, and uses a visual interface to show changes in states, actions, and rewards, helping learners build intuition before encountering formulas. The project covers core algorithms such as the Bellman equation, value iteration, policy iteration, Q-learning, and SARSA. Learners can switch algorithms, adjust parameters, and observe policy differences. The source code is hosted on GitHub and deployed on Vercel. The developer is soliciting community feedback, and the project is still in an early iteration stage.