Can Language Models Play the Wikipedia Game? A Navigation Capability Deep Dive

Testing AI Information Retrieval with the Wikipedia Game

As document collections grow too large to index one by one, language models are increasingly being used to support or even automate information retrieval. But when retrieval tasks require specialized linguistic or scientific knowledge, the difficulty rises sharply. A new study cleverly repurposes the classic "Wikipedia Game" as a quantifiable testing platform for evaluating how well language models navigate information retrieval scenarios.

The rules of the Wikipedia Game are simple: starting from one article, players can only follow hyperlinks on each page to eventually reach a target article. Despite its game-like appearance, the task actually tests a model's understanding of conceptual relationships — it must judge "which link from the current page gets me closer to the target" — logic that closely mirrors real-world retrieval in large-scale knowledge bases.

How the Study Designed Its Cross-Language, Cross-Domain Comparison

The research team cast a wide net. They compared two technical approaches — sentence embedding models and large language models (LLMs) — across five Wikipedia language versions and five academic literature datasets. This design ensures that conclusions aren't limited to a single language or domain, giving the findings much broader applicability.

To avoid the uncertainty of relying on third-party APIs, the team built their own large-scale local graphs from scratch — both the hyperlink network between Wikipedia pages and a paper citation network based on OpenAlex data. They also open-sourced the code for building and navigating these graphs, allowing other researchers to reproduce and extend the experiments without any external interface constraints. This "self-built graph + open-source tools" approach is a clear win for research reproducibility.

For baselines, the study introduced human players and TF-IDF (the classic term frequency–inverse document frequency retrieval method) as reference points. Humans represent intuitive semantic navigation ability, while TF-IDF represents classic non-neural retrieval methods — together they define the reference coordinates for evaluating language model performance.

Sentence embedding models work by mapping a piece of text to a vector in high-dimensional space, where semantically similar texts are closer together. In navigation tasks like the Wikipedia Game, an embedding model computes the vector similarity between the current page and the target page, greedily jumping to whichever neighboring node is "most similar" to the target. TF-IDF (Term Frequency–Inverse Document Frequency), by contrast, is a classic keyword weighting method: TF measures how often a word appears in the current document, while IDF penalizes words that appear frequently across all documents (like common function words). Multiplied together, TF-IDF highlights a document's characteristic terms. Since TF-IDF understands no semantics and only matches at the lexical level, its performance ceiling is naturally lower than embedding models or LLMs — in comparative experiments, it anchors the "minimum expected performance."

OpenAlex is a fully open academic metadata platform hosting over 250 million papers and their citation relationships, maintained by a nonprofit and freely available for full data snapshots. By building a citation network from OpenAlex — transforming "paper A cites paper B" into directed edges in a graph — the research team essentially replicated the Wikipedia hyperlink navigation logic in an academic literature retrieval context, testing whether the conclusions hold across domains.

Key Findings: Top Models Approach Human Performance, But Don't Surpass It

The most striking finding from the experiments: no model was able to significantly outperform humans. However, the best-performing models — GPT-o3 mini, GPT-4o, QwQ:32B, and DeepSeek^Ad-R1:70B — achieved success rates comparable to human players. This suggests that today's top-tier models have become quite capable at conceptual association and path planning, matching human intuitive navigation.

Notably, this list includes both closed-source GPT-series models and open-source representatives like QwQ and DeepSeek-R1, demonstrating that open-source models can now compete with top commercial models on these navigation tasks.

Another important observation is performance instability: model performance and rankings shift noticeably across different languages and topics. In other words, a model that excels on English Wikipedia doesn't necessarily transfer that advantage to other languages or specialized domains. This variability has direct practical implications.

Several factors explain why model performance fluctuates across languages and topics. First, uneven training data distribution: the vast majority of LLM pretraining corpora are English-dominant, with significantly lower coverage of other languages, meaning models have internalized conceptual relationships in non-English Wikipedia far less thoroughly than in English. Second, domain vocabulary sparsity: even in English, training data for highly specialized fields (such as niche academic areas) may be relatively scarce, degrading the quality of concept representations in the embedding space. Third, graph structure differences: Wikipedia editions vary enormously in page count and link density — English Wikipedia has over 6.5 million articles, while some language editions have only a few hundred thousand, and sparse link networks inherently make path planning harder for both humans and models. These factors combine to produce the "rankings shuffle when you change the language or domain" instability effect.

Implications for Practitioners

For developers and researchers deploying language models for information retrieval, this study sends several pragmatic signals.

Model selection cannot be one-size-fits-all. Since performance varies by language and topic, real-world testing in the target scenario (specific language, specific domain) before deployment is essential — you can't simply rely on rankings from general-purpose leaderboards.

The value of the Wikipedia Game as an evaluation paradigm is also highlighted here. It provides a unified framework for robust comparison across different technologies, languages, and domains. Compared to many closed benchmarks, this visual, interpretable task format makes it much easier to understand the quality of a model's decision-making as it "advances step by step."

Today's language models haven't yet dominated humans in these navigation tasks, but the gap has narrowed to the point of near-parity. As model reasoning capabilities continue to improve, this game may well be fully conquered by AI in the not-too-distant future.