RuneScape Bench: Testing AI Agents' Real Abilities with an MMO

While AI agent evaluation remains focused on web navigation, code generation, and office automation, some have set their sights on a virtual world that has been running for over twenty years—RuneScape. A video project called "RuneScape Bench: MMOs for Agents" appeared on Hacker News, proposing to use a massively multiplayer online role-playing game (MMORPG) as a testing ground for agent capabilities. The value of this idea isn't that it adds another leaderboard, but that it may redefine how we measure whether an agent is "smart enough."

Why RuneScape, and Not Just Another Web Task Set

Mainstream agent benchmarks are mostly built around deterministic tasks: clicking links, filling out forms, calling APIs, writing code that runs in a sandbox. These tasks have clear boundaries and immediate feedback, making them easy to score—but they also train agents into "instruction executors." RuneScape offers a different kind of environment: vague goals, open-ended paths, and delayed feedback.

In RuneScape, for a character to grow stronger, it might first need to chop wood to gather materials, then forge equipment, then grind monsters in a specific area—all while managing inventory space, assessing risk, and deciding whether to trade with other players. These steps have no standard answers, and a misjudgment at any step only becomes apparent dozens of steps later. This long-horizon, partially observable, and irreversible-state nature precisely corresponds to the core difficulties of real-world tasks. Using a game as a benchmark is essentially using a sufficiently complex simulated environment to approximate the uncertainty of real scenarios.

RuneScape is an MMORPG launched by the British company Jagex in 2001, still continuously updated and operated today, with cumulative registered accounts exceeding 300 million. Its most distinctive feature is a skill-driven progression system: players have no fixed class or level, but instead raise corresponding values by repeatedly practicing over twenty skills such as woodcutting, mining, smithing, and fishing, with skills forming dependency chains through materials and finished goods. This design makes "getting stronger" not a linear monster-grinding level-up, but a resource network that must be planned out on your own. Additionally, RuneScape's client has long been primarily click-based, with high information density but low automation, and the player community has developed a vast Wiki and guide ecosystem. For agent evaluation, this means the environment has both clearly queryable knowledge and demanding fine-grained execution operations, naturally creating a gap between "knowing what to do" and "being able to do it"—exactly the evaluation tension that traditional web task sets struggle to reproduce.

Three Technical Highlights of MMOs as Agent Benchmarks

Long-Horizon Planning and Memory Retention

RuneScape's skill system naturally demands long-horizon planning. If an agent wants to accomplish a composite goal, it must maintain memory of its own state, resource inventory, and map information across hundreds of operations, and dynamically adjust its plan. This contrasts sharply with most current benchmarks that only test single-step or short-chain tasks, and can effectively expose agents' "goal drift" problems under long contexts.

"Goal drift" refers to the phenomenon where an agent gradually deviates from its original objective during long-sequence execution: as context grows, the intent set early on is diluted by local feedback from intermediate steps, and the model begins optimizing for immediate small goals while forgetting global constraints. In environments like RuneScape, goal drift often manifests as the agent repeatedly gathering the same resource, looping on already-completed subtasks, or forgetting why it was heading to a particular map. Related is "memory decay"—when historical information exceeds the context window and is truncated or compressed, the agent's ability to track its own inventory, completed progress, and unfulfilled commitments drops significantly. Most current benchmarks have task chains of only a few to a few dozen steps, where goal drift barely surfaces; MMO tasks, by contrast, can span hundreds of steps, amplifying this failure mode to an observable and quantifiable degree—which is why long-horizon planning is listed as a separate evaluation dimension.

Exploration and Decision-Making in an Open World

RuneScape has no preset "walkthrough route," and quests often have multiple solutions. Agents need to actively explore, learn through trial and error, and recover from failure, rather than following a script written by the evaluator. This is closer to the real business situation of "nobody tells you what to do next," and better distinguishes agents with autonomous decision-making ability from those that only do pattern matching.

Economic Systems and Multi-Agent Interaction

As an MMO, RuneScape has a player-driven trading market. If multiple agents enter this world simultaneously, they will naturally produce behaviors such as resource competition, bargaining, and cooperation. This provides an experimental ground approaching a real society for multi-agent collaboration and game-theory research, with complexity and emergence far exceeding static multi-agent dialogue datasets.

"Emergence" in multi-agent research refers to behaviors exhibited by the system as a whole that cannot be directly derived from the rules of individual agents. When multiple autonomous agents share a market with limited resources and uniform rules, macro phenomena such as price fluctuations, division of labor, monopolies, and collusion arise spontaneously rather than being pre-programmed by designers. The value of such phenomena lies in their unpredictability: static multi-agent dialogue datasets typically only specify roles and topics, with highly controlled interaction outcomes; whereas player-driven economic systems introduce real scarcity and competitive pressure, and agent strategies are selected and evolve through repeated games with other agents. Researchers can thereby observe how cooperation is established, how trust collapses, and how individual rationality leads to collectively suboptimal outcomes—research subjects that are difficult to reproduce reliably in closed sandboxes.

Problems This Benchmark May Expose

Using a game as an evaluation benchmark is not without controversy. RuneScape's interface operations are cumbersome, with a large amount of time consumed by mechanical clicking and movement, which introduces noise unrelated to "intelligence," making evaluation results partly dependent on operational efficiency rather than decision quality. Additionally, game rules change with version updates, and the benchmark's reproducibility and long-term comparability require extra engineering maintenance.

A more practical issue is cost. Progression in RuneScape requires substantial online time; if an agent's task cycle is measured in hours or even days, the computational and time overhead of evaluation will far exceed traditional benchmarks. Whether the project team needs to design acceleration mechanisms or phased tasks will directly affect whether this benchmark can be widely adopted by the community.

Judging from the discussion heat on Hacker News, the project's appeal lies in pointing out a direction: agent evaluation is shifting from "can it complete the task" to "can it continuously make reasonable decisions in a complex, open, long-term environment." RuneScape is just a vehicle; the real question is whether evaluation designers are willing to give up their obsession with determinism and accept a messier testing environment closer to the real world.

"Reproducibility" is the cornerstone of benchmark research: the same agent evaluated repeatedly under the same conditions should yield comparable results. Game-based benchmarks face special difficulties in this dimension. On one hand, MMOs adjust skill values, map layouts, and quest flows with version updates, and old evaluation records may become completely invalid under new versions; on the other hand, the game world contains other real players and random events, making the same path difficult to replay precisely. Common mitigation measures include fixing client versions, using private server snapshots, and isolating random seeds and player interference—but these engineering investments significantly raise the maintenance threshold of the benchmark. Without unified version locking and replay mechanisms, results reported by different teams will be difficult to compare horizontally, and the benchmark will struggle to accumulate into community consensus.

Implications for Agent Research

If MMO benchmarks become a trend, the dimensions of agent capability evaluation may need to be restructured. Traditional task success rate is no longer the sole metric; planning efficiency, error recovery ability, resource management level, and interaction strategies will all enter the scoring system. This also imposes new requirements on model training: agents must not only learn to "answer," but also learn to "live in a constantly changing world."

RuneScape Bench is currently still an early exploration presented in video form, and its scale and standardization are not yet sufficient to be called a mature benchmark. But the questions it raises are sharp enough: when sandbox tasks have been saturated by agents, what should we use to measure true intelligence? Perhaps the answer is not in more elaborate question banks, but in a virtual world that has existed for over twenty years and is full of real players.

One-line Summary

RuneScape Bench proposes replacing deterministic task sets with an MMO to evaluate agents' real decision-making abilities in open, long-term environments.

Paragraph Summary

RuneScape Bench is an early project presented in video form, advocating the use of the twenty-plus-year-old MMORPG RuneScape as an evaluation environment for AI agents, replacing deterministic task sets with clear boundaries such as web navigation and code generation. The article points out that RuneScape's vague goals, open paths, delayed feedback, and irreversible states can effectively expose agents' shortcomings in goal drift, memory decay, and error recovery under long contexts; the player-driven economic system provides an experimental ground with emergence for multi-agent collaboration and game theory. At the same time, this approach faces practical challenges such as operational noise interference, reduced reproducibility due to version updates, and high evaluation time and computational costs. The author believes its true value lies in pushing evaluation from "can it complete the task" to "can it continuously make reasonable decisions in a complex, open environment."