Q2D-Web Dataset: Benchmarking Agent Query Reformulation Across 23,000 Real-World Searches
Q2D-Web benchmarks agent query reformulation using 23,000 real production searches across 10 languages.
Q2D-Web is an evaluation dataset designed to benchmark query reformulation capabilities in retrieval-based agents. It contains 23,000 privacy-scrubbed real production search records spanning ten languages and dozens of domains, collected over nine months. Unlike benchmarks relying on synthetic data, Q2D-Web reflects genuine user behavior in production settings. Its evaluation methodology focuses on an agent's ability to reformulate user requests into primary and support queries, with each query independently assessed for relevance — enabling fine-grained diagnosis of failures in intent understanding versus information-completion strategy.
A Overlooked Evaluation Gap
As retrieval-augmented generation (RAG) and agent systems continue to evolve rapidly, there has been a persistent lack of realistic benchmarks for objectively measuring an agent's ability to translate vague user requests into effective search queries. Most existing datasets are either limited in scale or rely on synthetic data, leaving a significant gap between lab conditions and the complexity of production environments.
Q2D-Web was built specifically to address this gap. Based on publicly disclosed information, it brings together 23,000 PII-free production search queries, spanning ten languages and dozens of domains, collected continuously over a period of nine months. This scale and diversity sets it apart from most test sets cobbled together under controlled laboratory conditions.
Three Defining Characteristics of the Dataset
Real Production Data, Not Synthetic Samples
The core value of Q2D-Web lies in its data source — it is drawn from real production search behavior, not manually crafted or model-generated queries. This means the dataset reflects the diversity, colloquial nature, and domain breadth of how users actually make requests in practice. For teams looking to evaluate agent performance in real deployment scenarios, this kind of data carries far more credibility than synthetic benchmarks.
Personal identity information was also stripped at the collection stage — a necessary and responsible approach given today's growing compliance requirements. This ensures the data is suitable for open research while avoiding privacy risks.
Breadth Across Languages and Domains
Covering ten languages and dozens of domains, Q2D-Web is naturally suited to testing a model's generalization capabilities. An agent that performs well only in English within a single domain will often reveal its weaknesses in the multilingual, multi-topic reality of the real world. The nine-month collection window also means the dataset captures search needs across different time periods and contexts, reducing the bias that comes from a single time window.
The Evaluation Logic Behind Agent Query Reformulation
What truly deserves attention is the evaluation methodology behind the dataset. In Q2D-Web's setup, an agent reformulates a user's original request into a primary query and a set of support queries.
This design reflects how modern retrieval-based agents actually operate: when faced with a complex or ambiguous user intent, a single query is often insufficient to retrieve complete information. The agent needs to decompose the request into one core query, supplemented by several additional queries to cover different facets of the needed information.
Even more significant is the evaluation granularity — each query is assessed independently, with its own relevance judgments. This approach avoids scoring the agent's overall output as a single block, and instead enables precise identification of issues: Does the primary query accurately capture the user's core intent? Do the support queries effectively fill information gaps — or do they introduce redundant or off-target noise?
Why Independent Evaluation Matters
Evaluating primary and support queries separately essentially breaks down the agent's "query planning" capability into quantifiable sub-tasks. It enables researchers to answer more specific questions: Does the agent fail because it misunderstands user intent, or because its information-completion strategy is inadequate? This kind of fine-grained diagnosis has direct practical value for iterating on retrieval-based agents.
Implications for Retrieval-Based Agent Development
Q2D-Web provides a more realistic measuring stick for the query reformulation (query-to-document or query reformulation) stage of retrieval pipelines. For teams building RAG systems, search agents, or multi-step reasoning retrieval pipelines, benchmarks like this offer value in several ways:
- Testing system robustness under production conditions using real, multilingual data;
- Precisely pinpointing weaknesses in query generation through independent query-level relevance judgments;
- Providing comparable quantitative evidence for cross-domain generalization capabilities.
It should be noted that the information currently available comes primarily from a brief project overview. Specifics about the dataset's construction process, annotation standards, and accompanying baseline model results have not been fully detailed in the available materials. Researchers intending to adopt this benchmark should still refer to its complete original documentation and evaluation protocol to accurately understand its applicable scope.
Summary
With its three defining characteristics — real production data, multilingual and multi-domain coverage, and query-level independent evaluation — Q2D-Web fills a meaningful gap in benchmarking agent query reformulation capabilities. It reflects a broader industry trend: evaluation is moving away from synthetic, single-language lab environments toward settings that more closely resemble real deployment conditions. For teams that take retrieval-based agent quality seriously, this kind of benchmark is well worth adding to the evaluation toolkit.
Related articles
Microsoft Issues Emergency Windows 11 Patch to Fix Bugs Caused by Its Largest-Ever Update
Microsoft pushed an emergency out-of-band Windows 11 update to fix bugs introduced by its largest-ever September Patch Tuesday, which fixed nearly 1,000 vulnerabilities but broke Hyper-V folder sharing.
Voodoo Dynamic Quant Goes Open Source: Using Gradient Descent to Optimize Model Quantization
Voodoo Dynamic Quant (VQ) is now MIT-licensed open source. It uses gradient descent to optimize per-tensor GGUF quantization, outperforming Unsloth Dynamic 3.0 at aggressive low-bit levels.
Perplexity Reveals: The Embedding and Ranking Model Serving Infrastructure Behind AI Search
Perplexity reveals the serving infrastructure behind its AI search embedding and ranking models, showing how retrieval quality shapes answer accuracy and speed.