Q2D-Web: Redefining Retrieval Evaluation Realism with 190 Million Web Pages

Why Retrieval Evaluation Needs Larger Corpora and Query Sets

Information retrieval benchmarks have long suffered from a quiet but serious flaw: test sets are too small and relevance judgments too shallow, meaning a system may retrieve the correct document yet still be penalized because that document was never labeled. These false negatives systematically underestimate retrieval quality and make meaningful comparisons between methods nearly impossible.

Building evaluations that reflect real-world conditions requires three things at once: a large enough document corpus, a sufficiently diverse query set, and deep relevance judgments for every query — not just a handful of positive labels, but coverage of all genuinely relevant documents. Only then can benchmark results faithfully represent how a system performs in an open web environment.

Q2D-Web's Core Scale

According to the information disclosed from this source, Q2D-Web is built around exactly these three dimensions. Its key statistics include:

• A corpus of 190 million (190M) web documents, approaching the scale of real-world web retrieval;
• 69,721 agent-reformulated queries covering a wide range of retrieval intents;
• An average of 99.6 positive relevance judgments per query in the combined set.

The most striking figure here is that average of 99.6 positive labels per query. Traditional retrieval benchmarks typically provide only single-digit relevant document annotations per query — Q2D-Web pushes that density to nearly one hundred, directly targeting the false negative problem. The denser the judgments, the lower the probability that a correctly retrieved document gets misclassified as irrelevant, and the more trustworthy the evaluation becomes.

What Agent-Reformulated Queries Mean

The queries in this dataset are produced through agent-based reformulation — a design choice worth unpacking. Query reformulation involves rewriting, expanding, or normalizing a raw query into a clearer form that better expresses the underlying retrieval intent. Using an agent to automate this process enables the generation of nearly 70,000 queries at scale while maintaining semantic quality.

This approach reflects a broader trend in retrieval evaluation: leveraging language models and intelligent agents to overcome the cost and scale bottlenecks of manual annotation. For a corpus spanning 190 million documents, manually constructing queries and judgments is practically infeasible — agent assistance becomes the key enabler for making deep relevance annotation viable.

Implications for Retrieval and RAG Research

A larger corpus, more queries, and denser positive judgments all point toward the same goal: making benchmark results more representative of real production environments. This matters especially for the rapidly evolving fields of retrieval-augmented generation (RAG) and vector search — if the evaluation benchmark itself carries systematic bias, conclusions about which models perform better may be equally unreliable.

Datasets like Q2D-Web provide value by offering a web-scale test bed with more thorough relevance coverage, helping researchers more accurately measure the recall and ranking capabilities of retrieval models.

It should be noted that this article is based on a brief disclosure from a single source. The complete construction methodology, specific benchmark results, and data availability of Q2D-Web are still pending confirmation from additional official materials.