Game Benchmarks: A New Approach to Evaluating Large Language Model Capabilities

When Games Become the Litmus Test for AI Capabilities

Among the many methods for evaluating large language models (LLMs), traditional benchmarks typically focus on standardized tasks like mathematical reasoning, code generation, and commonsense Q&A. These benchmarks—such as MMLU (Massive Multitask Language Understanding, covering multiple-choice questions across 57 subjects), GSM8K (a collection of grade-school math word problems), and HumanEval (code generation evaluation)—form the primary framework for assessing AI model capabilities today. However, a developer recently made an interesting observation on social media: "runescape bench is always aligned with the models i like the most."

This seemingly casual comment actually touches on a topic worth deeper reflection: non-standardized evaluation methods that closely mirror real-world scenarios may better reflect the alignment between a model's actual capabilities and user preferences than traditional benchmark scores.

What Is the RuneScape Benchmark?

RuneScape is a classic massively multiplayer online role-playing game (MMORPG), released by UK-based Jagex in 2001. It's one of the longest-running MMORPGs in internet history and was once certified by Guinness World Records as the largest free MMORPG. The game has over 20 years of accumulated content, including 28 trainable skills, hundreds of quest chains, a complex Grand Exchange economic system, and constantly updated game mechanics. Its Wiki contains tens of thousands of pages, and the community has produced an extraordinarily rich body of guides, theoretical calculations, and meta analyses.

Using RuneScape as an "exam" for AI models means requiring models to understand the game's quest systems, item crafting, skill progression, economic systems, and other multidimensional knowledge. The depth and breadth of this knowledge makes it ideal material for testing an AI model's information integration capabilities—models need not only to remember discrete facts but also to understand skill dependencies, item crafting paths, and game balance mechanics that change with version updates.

Why Game Knowledge Makes a Good Test for LLMs

Game-based benchmarks like this are valuable for several reasons:

• Long-tail knowledge characteristics: Many details in games constitute long-tail knowledge that only models which have truly "read and integrated" vast amounts of community content can answer accurately. Long-tail knowledge is a concept from information distribution theory, referring to knowledge fragments that appear infrequently but are enormous in total volume. In internet data, a few popular topics dominate most content (the head), while millions of niche domains each contain only small amounts of content (the long tail). For LLMs, head knowledge is reinforced repeatedly during training and is easy for models to master; but long-tail knowledge, due to scarce training samples, places higher demands on a model's memory efficiency and knowledge compression capabilities. A truly powerful foundation model should maintain reasonable accuracy on long-tail knowledge, reflecting the breadth of its training data coverage, the efficiency of its knowledge encoding, and its generalization ability to extract patterns from limited samples.

• Multi-step reasoning requirements: Many game-related questions (such as "what's the shortest path to complete a certain quest") require models to plan and perform multi-step reasoning, rather than simple retrieval. For example, quests in RuneScape often have prerequisite quest requirements, specific skill level thresholds, and complex steps requiring travel between multiple locations, posing a genuine challenge to a model's logical chain construction ability.

• Resistant to score gaming: Unlike public benchmarks such as MMLU and GSM8K, obscure game knowledge is unlikely to be incorporated into training data optimization targets on a large scale, making it a better reflection of a model's true generalization capabilities.

The Misalignment Between Subjective Preferences and Objective Scores

The author's mention of "always aligned with the models I like the most" reveals a long-standing phenomenon in the industry: models that score highest on leaderboards are not necessarily the ones that deliver the best experience in users' daily workflows.

Benchmark Contamination: A Fatal Flaw in Traditional Evaluation Methods

As model scale and training data expand, an increasing amount of public benchmark test data is being inadvertently or deliberately included in training sets, leading to "Benchmark Contamination." Benchmark contamination can occur through multiple pathways: training corpora containing publicly available benchmark questions, web-crawled data mixed with solution discussions, or even developers intentionally performing data augmentation targeting specific benchmarks. Since 2023, multiple studies have revealed the severity of this problem—for instance, one paper found that certain models' high scores on the GSM8K math test dropped significantly when questions were slightly rephrased, suggesting models may have memorized specific problems rather than truly mastering problem-solving abilities. Organizations like OpenAI and Google have begun disclosing contamination detection results in their technical reports, but the industry still lacks unified contamination detection standards.

A model might achieve high scores on standardized tests while performing mediocrely on real, novel tasks. Niche, vertical domain knowledge tests like RuneScape constitute a kind of "hidden exam"—they're harder to optimize against specifically, and thus can filter for models that truly possess solid knowledge foundations and reasoning capabilities.

The Rise of Personalized Model Evaluation

An increasing number of practitioners are building personalized private evaluation sets (private evals), using questions from domains they're familiar with to test models. Building private evaluation sets has become a popular practice among AI practitioners, with the core methodology being: select domains where the evaluator has expert-level judgment, construct 50-200 questions spanning different difficulty levels, and maintain standard answers. Well-known AI researchers like Simon Willison and Andrej Karpathy have publicly discussed their private evaluation methods. Some advanced practices include: setting up situational questions requiring multi-step reasoning, trap questions containing common misconceptions, and boundary questions requiring models to acknowledge uncertainty.

The advantages of this evaluation approach are:

1. Testers have full confidence in the answers and can accurately judge whether model outputs are correct or wrong;
2. Questions are close to real use scenarios, making evaluation results more practically relevant;
3. Private test sets won't leak, avoiding the risk of models "memorizing answers"—since the questions have never been published, any model's performance on them represents genuine generalization ability rather than the result of targeted training.

The expression "models I like the most" is essentially saying: models that pass one's personal real-world tests tend to also excel on these reality-adjacent, obscure evaluations.

Implications for LLM Evaluation Methodology

This brief tweet actually provides an important reminder for the entire AI community.

The Necessity of Diversified Evaluation

Single standardized benchmark scores are losing their discriminative power. As of 2024-2025, the saturation phenomenon in mainstream benchmarks has become quite apparent: on the MMLU benchmark, GPT-4-level models score above 86%, with gaps between multiple models narrowing to 1-2 percentage points within statistical noise; the HumanEval code generation benchmark has similarly hit a ceiling effect. This saturation has led to "benchmark inflation"—score improvements no longer correspond to user-perceptible capability improvements.

The industry has therefore begun exploring more challenging evaluation methods, such as GPQA (graduate-level Q&A), SWE-bench (real software engineering tasks), and various Agent evaluation frameworks, attempting to re-establish differentiation between models. Game knowledge, professional domain Q&A, open-ended creative tasks, and other scenarios may all become important supplements to next-generation model evaluation systems—they provide more diverse, more realistic, and harder-to-optimize evaluation dimensions for distinguishing actual capability differences between models.

User Experience Is the Ultimate Standard for Model Evaluation

Regardless of benchmark scores, models ultimately need to serve real users. A model that performs consistently, reliably, and in alignment with needs in users' daily tasks is far more valuable than a leaderboard champion. This also explains why Chatbot Arena (a human preference evaluation platform operated by LMSYS, which ranks models by having users blind-test and compare responses from two models and vote for the better one) has gained increasing recognition in the industry—it directly reflects user preferences in real interactions, rather than model performance on standardized questions.

"Aligned with the models I like the most"—this statement precisely emphasizes the simple but important principle that evaluation should return to real user experience.

Conclusion: Build Your Own AI Evaluation System

From the interesting entry point of the RuneScape benchmark, we can see a subtle shift happening in the field of AI model evaluation: moving from the "teach-to-the-test" approach of pursuing standardized benchmark scores toward personalized evaluations that are closer to real scenarios and harder to contaminate.

For developers and users, rather than blindly trusting leaderboard numbers, it's better to build your own "exam"—test models with questions you truly care about and truly understand. Specific practical suggestions include: choose 2-3 domains you're most expert in, prepare 20-50 progressive questions per domain ranging from basic to expert level, cover three types—factual Q&A, reasoning problems, and open-ended questions—and periodically run new models through them while tracking score trends over time.

You might find that the model best suited to you has always been hiding in those corners that nobody else thinks to test.