Training a Snake AI on Raw Board Input: Simple Design Beats Complex Feature Engineering

In reinforcement learning, feature engineering is often seen as key to better performance, but one developer's experiment suggests the opposite: using the most basic board input actually produced a stronger Snake agent.

Project Background and Training Configuration

The developer built a Snake engine in C++17, exposed Python bindings via pybind11, wrapped it as a Gymnasium environment, and spent about three days training the agent. The final model achieved an average score of 267 (out of 297) over 200 greedy-policy games, with 21% of games filling the entire board.

Training configuration:

• Algorithm: PPO + GAE, 32 parallel environments, rollout length 256, gamma 0.998, lambda 0.97, a small residual CNN as actor-critic.
• Observation: fixed 20x15 board (with walls), 6 one-hot channels representing head, neck, body, tail, food, and walls respectively. No snake-centered rotation, no body-order channel, and no auxiliary head.
• Actions: 4 absolute direction keys.
• Rewards: -0.01 per step, +10 for eating food, -50 for death, +100 for filling the board.
• Failure replay: 8 of the 32 environments restart from snapshots of the past 20 to 50 steps before a collision, with each replay capped at 200 steps.
• Total training steps: 30 million steps.

PPO and GAE: PPO (Proximal Policy Optimization) is a policy-gradient-based reinforcement learning algorithm that uses importance sampling and a clipped objective to improve sample efficiency while constraining policy update magnitude. GAE (Generalized Advantage Estimation) is an advantage estimation method that uses an exponentially weighted sum of temporal-difference errors to balance bias and variance, with its lambda parameter (0.97 here) controlling the trade-off. The combination is commonly used for continuous or discrete action space tasks such as Snake.

Residual CNN: Residual networks (ResNet) use skip connections to mitigate vanishing gradients in deep networks, making training more stable. Here it is used as the actor-critic network, outputting action probabilities and value estimates respectively. A small residual CNN can effectively extract spatial features from the board.

One-hot channels: Each cell's state on the board is encoded as an independent binary channel (e.g., head, body, food), avoiding spurious ordinal relationships introduced by numerical values and allowing the convolutional network to clearly distinguish different entities.

Key Findings: Simple Input and Critical Parameters

Gamma's Impact Far Exceeds Expectations

In early experiments, merely changing gamma from 0.995 to 0.998 raised the average score from 107 to 150 with no other parameter changes. This shows that in long-sequence decision tasks like Snake, the discount factor is extremely critical for value propagation.

Discount factor gamma: In reinforcement learning, gamma (between 0 and 1) determines the present value weight of future rewards. The closer gamma is to 1, the more the agent values long-term returns; the smaller it is, the more short-sighted. In Snake, eating food and eventually filling the board require long-sequence decisions, so a higher gamma (e.g., 0.998) can more effectively propagate distant value signals and prevent the agent from ignoring long-term gains due to short-term penalties (-0.01 per step). This explains why adjusting gamma alone can greatly improve performance.

Less Input Engineering, Better Results

The developer tried more complex input designs: a snake-centered rotated view, body-order channels, and an auxiliary danger prediction head. But the plain fixed-board agent won at a comparable stage with a score of 192.8 versus 178.5, and learned a clean serpentine sweeping strategy on its own. This challenges the intuition that "more features equal better performance"—at least in Snake, letting the network learn spatial relationships from raw state seems more effective.

Details of Collision Rules and Replay Mechanism

Initially, the snake's head could move into a cell the tail was leaving, since the tail moves away in the same step. This meant the body directly ahead was sometimes fatal and sometimes safe, and the agent had to distinguish the two from subtle input differences. Changing the rule so that all body cells (including the tail) count as collisions improved training.

The failure replay mechanism only worked when measured in steps rather than games. The first version replayed a quarter of new games, but games lasted thousands of steps, so replay data was less than 1% of total training. After adjustment, replay truly took effect.

Counterintuitive Phenomena and Open Questions

The Agent "Expects Failure" Yet Wins

The most surprising finding: the agent's value estimate just before winning is typically -30 to -50. In other words, it completes filling the board while "believing it will lose." The developer speculates that failure replay only stores positions from failed games, so the crowded boards it practiced on almost all ended in collisions, leading to pessimistic value estimates.

Pessimistic bias in value estimation: The value function (critic) learns state values from historical experience. When failure replay contains only failed games, the values of those states are repeatedly updated to negative (collisions yield -50 reward), causing systematically low value estimates for crowded boards. However, the policy (actor) may still find winning paths because policy updates depend on the advantage function (actual return minus value estimate); a pessimistic value makes the advantage positive, encouraging exploration. This separation between value and policy is not uncommon in deep reinforcement learning but requires careful calibration to avoid training instability.

Limitations of the Auxiliary Danger Head

The separately tested auxiliary danger head version achieved 99.7% recall but still missed a rare straight-line configuration, causing confident collisions. And practicing with late-game snapshots (to correct value estimates near the endgame) caused performance to collapse: evaluation scores dropped from 190-230 to 40-130, and an early-game loop (snake circling until death, never seeking food) reappeared.

According to the developer's tests, most remaining failures come from free space splitting into disconnected regions late in the game. How to calibrate value estimates at the end of long episodes without breaking overall stability remains an open problem.

Practical Takeaways

This project offers several reusable lessons: the discount factor has extremely high priority in long-sequence tasks; simple observation design may outperform handcrafted feature engineering; failure replay needs coverage measured in steps; and there is a delicate trade-off between value calibration and training stability.

The project code and best model weights are open-sourced on GitHub (Partha11/hatchling), with documentation available online, suitable for developers who want to reproduce or experiment further.

One-line Summary

Developer trained a Snake agent on raw board input, finding simple observations beat complex feature engineering and revealing the key roles of gamma and failure replay.

Paragraph Summary

This article presents a Snake reinforcement learning project trained with C++ and PyTorch. The developer used PPO and a residual CNN with only 6 one-hot channels to represent board state, achieving an average score of 267 (out of 297) after 30 million training steps. Experiments show that changing gamma from 0.995 to 0.998 raised the score from 107 to 150; simple fixed-board input (score 192.8) beat the complex feature-engineered version (178.5). Failure replay coverage must be measured in steps, otherwise data share is under 1%. The agent often has negative value estimates before winning, exhibiting "expecting failure yet winning." Practicing with late-game snapshots caused performance collapse, and value calibration remains an open problem. Project code is open-sourced.