Can a Multimodal Decision Model Play World of Warcraft? The SageCraft Open-Source Experiment

A Crazy Idea Born in the Bathtub

13 days ago, a developer had a thought while soaking in the tub: could a multimodal decision model play World of Warcraft? The answer turned out to be pretty interesting — "kinda."

That idea eventually became a project called SageCraft — the first harness (agent framework) built around the Sage decision model. The goal was clear and demanding: play WoW entirely through screenshots and keyboard input, without modifying any game files or installing any add-ons. The whole system has just three components — the Sage model, the harness framework, and simulated mouse/keyboard input.

The developer openly admits to being a co-founder at Levanto, the company behind Sage. The project was partly a stress test of their own model, and partly an exploration of whether a dedicated decision model — beyond the well-known LLM "computer use" capabilities — could hold its own in real-world tasks.

Technical Architecture: Screenshots In, Keystrokes Out

SageCraft's runtime logic isn't overly complex, but executing it well requires serious engineering trade-offs. The entire system runs on macOS in a closed loop:

1. The harness captures game screenshots and high-level gamestate
2. It reads HUD (Head-Up Display) information and compiles a list of currently available actions
3. This data is packaged and sent to the Sage model for decision-making
4. The harness validates the model's choice, then executes the corresponding keypress or click
5. Repeat

The core design philosophy here is "the harness handles perception and execution; the model handles decisions." The developer intentionally feeds the model a list of available actions rather than letting it generate arbitrary inputs from scratch — this improves reliability while implicitly constraining the model's autonomy.

Notably, the author believes this harness is generalizable — theoretically, it could be swapped out for any multimodal-capable decision model, not just Sage. That makes the project more than a product demo; it's closer to a reusable experimental paradigm.

HUD (Head-Up Display) refers to the information layer overlaid on the main game view, typically including character health, mana, minimap, skill bars, and experience bars. In SageCraft's design, the harness must parse HUD elements from screenshots and convert them into structured gamestate data before passing everything to the Sage model. This is non-trivial in practice — WoW's HUD layout is complex, coordinates shift across different resolutions and UI scaling settings, and displayed elements vary between combat and non-combat states. This is one key reason the author needed to manually fix harness issues between sessions: the robustness of visual parsing directly determines whether the entire closed loop can run stably.

The Scorecard: A Dwarf Reaches Level 5

The quantitative results are both compelling and refreshingly honest:

• The in-game dwarf character reached Level 5
• The run consumed 4,573 decisions
• Spread across 32 game sessions
• With a total of 5 hours and 29 minutes of actual playtime

These numbers reveal the true meaning behind that word "kinda." The author supervised the entire run and manually fixed harness issues between sessions. In other words, this was not a fully autonomous, hands-off AI player.

That said, one highlight stands out: the final 18 minutes — from late Level 4 all the way to Level 5 — were completed with zero human intervention. This shows that when the harness logic is solid enough and the environment is stable enough, the decision model can genuinely sustain gameplay on its own.

The Real Challenge: Where Does the Logic Belong?

The most valuable discussion point in this project is actually the question the developer openly posed to the community: Where should the line be drawn between hard-coded harness logic and model-driven autonomous decision-making?

This is an unavoidable core tension for anyone building agent harnesses. Push too much logic into the harness, and the model degrades into a mere executor with no generalization ability. Leave too many decisions to the model, and reliability collapses — with decision counts exploding (4,573 decisions for 5 levels says it all).

SageCraft's middle-ground approach — providing an action list plus harness-level validation — is currently one of the more mainstream patterns in agent engineering. But in an environment like WoW, with its enormous state space and real-time demands, there's still no standard answer for where that boundary should optimally sit.

This question is often framed in AI agent research as the "tool granularity" or "action abstraction" problem. If the harness provides highly abstract atomic actions (like "attack nearest enemy" or "move to target"), the model's decision burden decreases but its generalization space shrinks. If actions are broken down to individual keypresses, the model needs a much longer planning horizon to produce meaningful behavior sequences. Projects like OpenAI's Operator and Anthropic's Computer Use face the same trade-off. The current industry lean is toward providing "composable macro actions" at an intermediate layer — where the harness guarantees execution reliability and the model focuses on intent-level scheduling. SageCraft's action list approach aligns with this trend.

Why This Experiment Is Worth Paying Attention To

Stripped of the "playing games" entertainment framing, SageCraft actually touches on several cutting-edge questions.

First, the capability overlap between decision models and large language models. Over the past year, LLMs' "computer use" capabilities have been emphasized repeatedly. The author's implicit reminder: dedicated decision models can do this too, and may even be more efficient in specific scenarios. That has real implications for understanding the applicable boundaries of different model architectures.

Second, the generality of pure vision + keyboard control. No memory reading, no plugins, no game file modifications — just screenshots and simulated input to operate a complex 3D game. This "play like a human" paradigm is precisely one of the goals that general-purpose agent research is chasing.

Third, the value of open-sourcing. The author published the harness code and recorded a build video, giving other researchers a directly reproducible and improvable foundation. He even half-jokingly noted that Blizzard might want to know that — beyond LLMs — decision models are also capable of doing this.

Decision models differ from large language models (LLMs) in their core training objective: LLMs are trained primarily on language prediction, and decision-making emerges as a byproduct. Decision models, by contrast, are typically trained via reinforcement learning or preference-based optimization methods, with the explicit goal of selecting optimal actions in an environment — much closer to the nature of sequential decision problems. Levanto's Sage falls into this latter category. Such models tend to be more efficient than general-purpose LLMs in terms of action selection latency and token consumption, though they may fall short of models like GPT-4o or Claude on open-ended language understanding and complex reasoning. One of SageCraft's contributions is providing a quantifiable experimental scenario that makes the capability boundaries between these two model types more clearly visible.

The Honest Reality

It's important to be objective: the current SageCraft is nowhere near "an AI independently conquering World of Warcraft." Full human supervision throughout, manual harness fixes between sessions, 5 hours to reach Level 5 — all of this signals that it's still a very early-stage proof of concept. That "kinda" is an honest, measured self-assessment.

But that restraint is exactly what makes the project credible. There's no hype, just real data showing both the potential and the limitations of multimodal decision models in complex interactive environments. For developers focused on shipping AI agents in the real world, small and honest experiments like this are often far more valuable than grand narratives.