Hybrid AI Agent Clears Diablo I Ironman Mode, Reaches Level 9

A hybrid AI agent has completed dungeons 1 through 9 consecutively in Diablo I's Ironman mode. This is the fourth progress update for the project, with developer rouming publishing full technical details on Reddit.

What Is Ironman Mode

The project's core goal is to have the agent kill the final boss Diablo in a single run without human intervention. Ironman mode means: no town portals, only one life, dropped items stay permanently on the current floor, and all 16 dungeon levels must be cleared in sequence.

This choice wasn't just about increasing difficulty. The developer admits in the original post that Ironman mode is actually "the easiest mode to implement" — no town portals means no town navigation code to write, one life means no ambiguity about success or failure, the hero just goes down, and the agent never needs to decide whether to backtrack. Fewer decisions means less room for error.

Another key setting is disabling the quest system. In Diablo I, some quests rewrite dungeon layouts, such as the Chamber of Bone on level 6 and the Halls of the Blind on level 7, both generating doors that require completing a quest to open, with the stairs conveniently hidden behind them. Disabling quests keeps dungeon layouts predictable and stairs always reachable. The developer also candidly admits: no Butcher definitely made the survival data look much better.

Diablo I's quest system isn't just side content — it directly rewrites dungeon terrain generation. Take the Chamber of Bone on level 6: players must find and read a specific book to open the door to the next level; the Halls of the Blind on level 7 similarly requires killing a specific monster to obtain a key. For human players this is just part of the flow, but for an agent, quest trigger conditions, item acquisition paths, and door lock states are all additional hidden variables — if a quest item doesn't spawn in a given random generation or the model misses an interaction, the entire run gets stuck before the stairs. With quests disabled, the dungeon generator only uses basic room and corridor templates, stair connectivity is guaranteed, and the agent's exploration strategy can be simplified to a pure geometric problem. The Butcher is a fixed elite monster on level 2 with extremely high damage and a charge attack, making him the primary cause of death in early floors; removing him along with quests does smooth out survival data for the first four floors, but at the cost of the agent losing training opportunities against high-pressure melee enemies.

Hybrid Architecture: Algorithms for Determinism, Neural Networks for Uncertainty

The project's most core design philosophy isn't "use AI for everything" but "write algorithms where you can, and only hand truly difficult parts to neural networks."

The Supervisor is a state machine. Most of the time it runs in DUNGEON mode, with the reinforcement learning model controlling decisions. When the step budget for a floor is exhausted or enough monsters are killed, the Supervisor switches to PATHFIND mode, using BFS pathfinding to the most recently seen stairs. The model itself never sees stair triggers — they're disguised as walls to prevent the model from accidentally triggering them in combat. Whether to descend is decided by the Supervisor.

The reinforcement learning model handles all decisions in the dungeon: where to move, when to attack, which corridor to explore, whether to open doors/chests/sarcophagi, when to pick up items. Potion usage is also decided by the model — it observes current small/large/rejuvenation potion counts and issues health or mana recovery commands when it judges the timing is right.

In terms of network structure, a CNN processes a 21x21 local observation window, and an LSTM carries memory across steps. Three attention modules are stacked on top of the CNN: SelfAttention, MemoryCrossAttention, and FiLM. Ablation experiments in Part 3 showed that FiLM does nearly all the effective work, while the other two are essentially dead weight in the trained model — a result that surprised the developer.

The project trained two independent models: one for levels 1-4 (Cathedral), another for levels 5-8 (Catacombs). The two areas differ in geometry, monster pools, and hero attributes, and a single model covering all floors performs worse. Level 9 and beyond fall back to using the Cathedral model, since only about 3% of runs reach there, making separate training not worthwhile.

FiLM (Feature-wise Linear Modulation) is a conditional normalization technique originally used for visual reasoning tasks. It uses an auxiliary network to generate a set of scaling coefficients γ and offset coefficients β based on current context, then applies an affine transformation to each channel of the main feature map: output = γ · feature + β. In this project, FiLM allows the model to dynamically reweight CNN-extracted spatial features based on scalar states like current health, mana, and potion counts — essentially telling the model "now you should focus more on enemy positions or escape routes." SelfAttention and MemoryCrossAttention serve to let spatial features reference each other internally and to let the current frame query historical information stored in the LSTM. Ablation experiments showed the latter two contribute nearly zero after training completes, suggesting that the 21x21 local window already contains sufficient spatial relationships for decision-making, and the LSTM's hidden state already indirectly influences feature extraction through FiLM's modulation channel, making additional attention paths unnecessary. This result also suggests: in scenarios with small observation windows and limited state dimensions, complex attention modules may be replaceable by simpler conditional modulation.

Attribute Strategy: Dexterity Rush

Each level-up grants 5 attribute points. The agent adopts a "dex-rush" strategy — a technique Diablo I players have used for 30 years: prioritize dexterity, defer other attributes.

The reason is straightforward: dexterity feeds directly into the attack rating formula, determining whether attacks hit. Levels 1-4 enemies have low armor, so stacking dexterity lets the warrior hit more accurately, ending fights faster and taking less damage. The developer tried strength-priority and balanced allocation, and dex-rush won out. This strategy is hardcoded and doesn't need to be learned.

Equipment Management: From Weighted Sums to Full Combat Formulas

The equipment system underwent the most rewrites. The initial version scored items using weighted attribute sums — dexterity weighted higher than vitality, and so on. This worked on early floors but started failing in ways that were hard to debug on deeper floors, because Diablo's attribute interactions aren't linear.

The solution was translating the original DevilutionX engine code for calculating attack rating, damage, and armor values into Python, producing a function that computes the hero's expected combat efficiency against the average monster on the current floor:

(hero_hp * hero_damage_per_hit * hero_hit_chance) / 
(monster_hp * monster_damage_per_hit * monster_hit_chance)

Average monster attributes per floor are measured from the real engine across 200 seeds. When a new item appears, the agent virtually equips it, recalculates this value, and compares it with the current value. If the number goes up, it's worth equipping. The key insight: a +10 dexterity ring might be excellent or worthless depending on whether hit chance has already hit the 95% cap, whether the hero carries a shield, and what the current character level is. Weighted sums can't perceive these; the full formula can.

Unidentified items are a special case. The agent can directly equip items based on visible attributes (base armor value, raw damage range), but items might be cursed. When equipping an unidentified item, the previous equipment is kept in inventory as backup. If the new item reveals cursed attributes after identification, the agent discards it and reverts to the backup equipment — the hero is never caught bare-chested in combat.

DevilutionX is an open-source remake engine for Diablo I, reimplementing the original game's combat, item, and monster logic in C++ while guaranteeing behavior consistent with the original. The developer translated its attack rating, damage, and armor calculation code line by line into Python, essentially embedding a lightweight game rules simulator inside the agent. The key advantage of this approach is verifiability: hit chances and damage values computed by the simulator can be checked item by item against real game frame data, and any deviation can be traced to a specific formula term. Every quantity in the expected combat efficiency function has a clear source — hero_hp is the current health cap, hero_damage_per_hit is the expected per-hit damage after strength, weapon damage, and skill modifiers, hero_hit_chance is determined by the ratio of attack rating to monster defense rating; the denominator similarly corresponds to the monster side. Averaging across 200 seeds eliminates random fluctuations in monster combinations from single generations, making each floor's "average opponent" a stable comparison baseline. This way, item scoring no longer relies on manual tuning but is derived directly from game rules, and doesn't require redesigning weights when changing floors or character levels.

Reward Shaping: Event-Table-Driven Iteration

Training started with sparse rewards — only terminal outcomes triggered rewards, and the model figured out the rest on its own. This approach went far, then stalled. Subsequent shaping was mostly trial and error.

After each training run, all action events were exported to tables: attacks per turn, items collected, wasted actions, etc. This approach is simple but extremely useful. If the model repeatedly casts fireball eight times with zero mana, the table immediately tells you, and the fix is obvious. If wasted attacks drop after increasing penalties, shaping is working; if not, something else is going on.

From the event frequency table: movement penalties account for 48.3% of total events, exploration tiles 21.7%, monster attacks 14.5%. Some adjustments worked — penalizing phasing behavior near monsters without targets was effective. The group combat amplifier was tested and ultimately abandoned: rather than adjusting rewards, lowering hero attributes during training naturally causes the hero to take more damage and learn to lure enemies.

The hardest part was balancing movement penalties and exploration rewards. After introducing exploration rewards in version 16, incremental calculation still made "standing still" slightly better. Both values had to be adjusted simultaneously before the agent started genuinely exploring new tiles. The last usable environment version was 17.

Monte Carlo Simulation: Calibrating the Training Curriculum

To set the right training curriculum, you need to know what real runs look like on each floor. How much experience does a warrior earn on levels 1-4? What level are they typically at when entering level 5? What equipment is realistically available?

The developer wrote a simulator (scripts/diablo-sim.py in the repo) using Markov chains to run a warrior through all 16 levels: sampling experience from kills measured in the real engine, applying level-up formulas, tracking equipment transfer between floors. After enough simulations, you get the distribution of hero states when entering each floor. Training at the wrong character level produces agents that work in training but collapse in real runs.

Markov chains here refer to a state transition model: treating "current floor, hero level, equipment strength, remaining potions" as states, and "kill monsters for experience, pick up items, level up, descend" as transition events, with each step's transition probability estimated from real engine measurement data. Since Diablo I's floor generation and experience distribution are random, a single simulation path has no reference value; thousands of runs are needed to get the distribution of hero attributes when entering each floor. The specific question the developer cared about: if hero starting level is set too low during training, the model learns risky tactics for punching above its weight and dies immediately in real runs when equipment luck is bad; set too high, and the model doesn't learn necessary conservative behavior. Using the simulator's distribution to set the training curriculum is equivalent to having the agent learn from "typical starts" rather than "ideal starts." scripts/diablo-sim.py itself doesn't participate in actual combat — it's just a calibration tool — but without it, systematic bias emerges between the training environment and the real environment.

Current Bottleneck and Next Steps

The developer admits in the original post to having hit a bottleneck. The past few months have been spent trying to improve survival rates on levels 5-8, but the training curve has flattened. He's unsure whether this is the ceiling of the current architecture or something he's missed.

The Catacombs are the bottleneck — only 3% of runs reach level 9, which is a result of levels 5-8 being too hard, not levels 9-12 being impossible. The most direct workaround is allowing town portals and trading: better equipment and more healing potions would help a lot. But that means writing town navigation and trading code, and the developer feels it's a bit like abandoning the original Ironman spirit.

Repository: https://github.com/rouming/DevilutionX-AI

Based on the Reddit source and developer rouming's project updates: this series of experiments demonstrates a pragmatic hybrid AI design approach — assigning deterministic tasks to traditional algorithms and leaving uncertain decisions to reinforcement learning — which may be an engineering paradigm worth watching in game AI and broader agent research.

One-line Summary

A hybrid AI agent clears Diablo I Ironman mode levels 1-9, demonstrating an engineering paradigm of dividing labor between algorithms and reinforcement learning.

Paragraph Summary

Developer rouming has published the latest progress on a hybrid AI agent in Diablo I's Ironman mode: the agent has cleared dungeons 1 through 9 consecutively, bringing it closer to the final boss Diablo. The core design assigns deterministic tasks to traditional algorithms and uncertain decisions to reinforcement learning — a Supervisor state machine switches between DUNGEON and PATHFIND modes, while a CNN+LSTM+FiLM network handles combat and exploration decisions. Equipment management evolved from weighted attribute sums to a full combat efficiency formula based on the DevilutionX engine, with a cursed-item fallback mechanism for unidentified gear. Training iteration relies on event frequency tables and per-floor statistics, with Monte Carlo simulation used to calibrate the training curriculum. The current bottleneck is levels 5-8 in the Catacombs, where only 3% of runs reach level 9.