Tencent DiscoBench: The First Benchmark for Evaluating LLM Search Agents' Ambiguity Clarification Capabilities

The Hidden Weakness of Deep Search Agents

Over the past two years, LLM-driven search agents have advanced rapidly. A Deep Search Agent is an autonomous information retrieval system built on large language models (LLMs), whose core capability lies in extending traditional single-shot retrieval into multi-step, multi-tool-invocation reasoning chains. From a systems engineering perspective, a deep search agent is essentially a "Perceive-Plan-Act Loop" system—highly isomorphic to autonomous agent architectures in robotics, with the only difference being that the "tools" at the execution layer are replaced from physical sensors to API calls. Such architectures typically consist of three layers: an intent understanding layer (converting natural language queries into executable plans), an execution scheduling layer (managing tool-invocation sequences and resource allocation), and a result integration layer (fusing multi-source information into coherent output). Open-source frameworks like LangChain and LlamaIndex are mainstream toolchains built along this architectural line of thinking.

It's worth adding that the direct precursor of this architectural paradigm was the wave of "tool-augmented language model" research that emerged around 2022. Google's proposed ReAct (Reason+Act) framework was the first to systematically interweave and fuse chain-of-thought reasoning with tool invocation—before each action, the model first outputs explicit reasoning steps (Reason), then performs a concrete operation (Act), forming an alternating "reason-act-observe" loop that laid the foundation for the modern search agent paradigm. After LangChain was open-sourced at the end of 2022, it quickly became the de facto standard for building agent systems, and its AgentExecutor loop is precisely the engineering implementation of this paradigm.

Unlike early keyword search or vector retrieval, deep search agents can autonomously plan retrieval strategies, parse web content, integrate cross-source information, and arrive at final answers through iterative reasoning.

The rise of this technical paradigm rests on three key technical pillars: Reinforcement learning (especially RLHF and RLVR) enables models to learn from feedback signals when to invoke tools and when to stop searching. Among these, RLHF (Reinforcement Learning from Human Feedback) was systematically applied by OpenAI in InstructGPT, training reward models through human preference signals; RLVR (Reinforcement Learning with Verifiable Rewards) relies on objectively verifiable tasks (such as math problems or code execution) to provide reward signals, a path adopted by models like DeepSeek-R1. Tool-invocation capability (Function Calling) allows LLMs to dynamically call external tools—search APIs, code interpreters, browsers, etc.—during reasoning. Since GPT-4, Function Calling has become a mainstream interface standard, allowing models to "pause" mid-reasoning to call external APIs, then inject the results into the context to continue reasoning. Chain-of-Thought reasoning lets the model explicitly output intermediate reasoning steps before and after each tool call, forming a traceable decision path. Together, these three enable LLMs to dynamically adjust subsequent search directions after obtaining intermediate information, just as a human researcher would. Representative products include OpenAI's Deep Research, Google's Gemini Deep Research, and Perplexity Pro. From traditional static text retrieval to deep search agents capable of autonomously browsing the web and completing multi-step reasoning, vendors like OpenAI and Google are all actively positioning related products. However, such systems have an obvious yet often overlooked limitation: they assume the user's initial query is complete and unambiguous by default.

This is a huge gap from real-world usage scenarios. In reality, the queries users provide are often vague or even fragmented. For example, a user searches for "the award status of the director of last year's blockbuster film" but doesn't specify which region's blockbuster. If the system defaults to retrieving domestic films, all subsequent reasoning steps will go astray, ultimately wasting substantial computing resources while producing the wrong answer.

It is precisely to address this pain point that Tencent's Hunyuan team, in collaboration with Tsinghua University, released DiscoBench—the first benchmark specifically designed to evaluate search agents' dynamic ambiguity clarification capabilities. It examines only one core question: can a search agent proactively identify ambiguity and ask the user for clarification?

DiscoBench's Core Innovation

Compared with existing benchmarks, DiscoBench's biggest difference lies in this: it treats ambiguity as a phenomenon that dynamically emerges during the multi-step search process, rather than as a static attribute of a single query.

Query Ambiguity is a classic problem in natural language processing, tracing back to early information retrieval (IR) research. Researchers typically divide ambiguity into three levels: Lexical Ambiguity is the most basic level, referring to the same word having different meanings in different contexts, such as whether "Apple" refers to the fruit or the company; Semantic Ambiguity occurs at the syntactic level, where the same sentence structure can correspond to multiple semantic interpretations; Pragmatic Ambiguity depends on the speaker's intent and contextual background, and is the type most difficult for systems to automatically identify. In Chinese contexts, due to the broad reference scope of pronouns, the high proportion of polysemous characters, the extensive reliance on subject-omitting expression habits, and pragmatic conventions dependent on shared cultural background, the density of pragmatic ambiguity is far higher than in English—this structural difference makes ambiguity benchmarks specifically oriented toward Chinese scenarios of exceptionally high reference value to domestic LLM teams, and is also a key motivation for DiscoBench's focus on Chinese scenarios. In the question-answering domain, the AmbigQA dataset (2020) is an important milestone; based on the NQ dataset, it systematically annotated ambiguous questions with multiple plausible answers, advancing ambiguity-aware QA research.

Previous related benchmarks each had limitations. GAIA, BrowseComp, and others only provide unambiguous queries and merely test the model's multi-step reasoning ability, completely ignoring the ambiguity problem. Meanwhile, datasets specialized in ambiguity handling, such as AmbigQA, are static QA scenarios that do not simulate the dynamic search interaction process—they assume ambiguity is fully presented at the time of questioning and cannot simulate the dynamic scenario of "gradually discovering ambiguity during the search process."

DiscoBench's approach is: set up ambiguity checkpoints within a complete multi-step search chain. The model needs to proactively discover during the search process that some intermediate node lacks sufficient information, then proactively ask the user, rather than guessing or selecting options. For example, when the search reaches a certain step and multiple candidate entities meet the criteria, ambiguity arises at that point, and the model must judge for itself whether to confirm with the user. This design more closely matches real-world scenarios—a user's ambiguity is often not fully exposed from the start but discovered halfway through the search when the information proves insufficient.

How the Dataset Is Constructed

The entire dataset construction is divided into two stages, adopting a construction paradigm known in NLP as "Controlled Perturbation"—precisely inserting interference factors that satisfy specific conditions on the basis of a known correct reasoning chain. The advantage is that it can precisely control the type and difficulty of ambiguity while preserving the naturalness of real-domain data. This method has extensive precedent in NLP data construction, with typical cases including Stanford's SQuAD adversarial dataset and ANLI (Adversarial NLI) in the natural language inference domain. Its core idea is to preserve the semantic structure of real corpora while injecting controllable variables only along specific dimensions, thereby achieving a precise probe of a model's specific capabilities. Compared with directly collecting naturally ambiguous queries, this approach offers greater controllability; compared with purely manual construction, it offers greater linguistic naturalness.

The first stage prepares seed data: the team collected multi-hop QA samples from 11 real-world domains including film & TV, sports, gaming, academia, and finance. None of the questions can be answered by common sense; all must be resolved through external retrieval, and the answers are objectively verifiable.

Multi-hop Question Answering refers to a type of question that requires traversing multiple intermediate reasoning steps or multiple knowledge sources to arrive at an answer. For example, "What is the GDP of the native-language country of the most recent Nobel Prize in Literature winner?" requires first looking up the winner, then their native language, then that country's GDP, forming a "hop chain." Multi-hop reasoning is the core capability that distinguishes deep search agents from simple retrieval systems, but precisely because the reasoning chain is longer, an information deviation at any intermediate node will lead to an incorrect final answer. Understanding this risk mathematically: if the accuracy of each step in the reasoning chain is p, then the overall accuracy of an n-step chain is theoretically about p^n—even if single-step accuracy is as high as 90%, after 5 steps the overall accuracy drops to about 59%. This mechanism is called "Error Cascading" in the information retrieval field, and is a core engineering challenge facing deep search agents. It is also the fundamental reason for the phenomenon in DiscoBench where "checkpoint pass rates are far higher than final accuracy."

To address the error cascading problem, the industry has developed several mitigation approaches: Self-Consistency samples multiple reasoning paths in parallel and uses majority voting to reduce the impact of single-path errors; Process Reward Model (PRM) provides reward signals at each intermediate step rather than only scoring the final answer, with OpenAI's "Let's Verify Step by Step" being a representative work in this direction; Tree of Thoughts expands the reasoning space into a tree structure, exploring multiple paths via breadth-first or beam search. However, the common premise of these methods is that the reasoning starting point is clear—under ambiguous scenarios, this premise itself does not hold, which is precisely the fundamental reason why ambiguity clarification capability is irreplaceable.

The second stage injects ambiguity into the determined multi-hop reasoning chain. Each ambiguity point must satisfy three conditions: the target entity has similar alternative options, the reasoning chain can still proceed after relaxing constraints, and a single clue from the user is sufficient to distinguish them. This ultimately generated 211 test samples containing 463 ambiguity instances.

Four Ambiguity Types and Evaluation Dimensions

The 463 ambiguity instances cover four types, essentially encompassing the major ambiguity scenarios in real-world searches:

• Entity Ambiguity: Multiple entities fit the same description. For example, "the sci-fi work of a director surnamed Zhang"—both Zhang Yimou and Zhang Fan fit.
• Version Ambiguity: Different states across time or versions. For example, "When was the voice progress adjustment feature launched in WeChat?"—the timing differs across system versions.
• Standard Ambiguity: The evaluation standard is unclear. For example, "the highest-grossing film"—is that global box office or mainland China box office?
• Factual Error Ambiguity: The user's description itself is inconsistent with facts. For example, "The Wandering Earth starring Lu Han"—the factual error must first be clarified.

The evaluation framework is specially designed for multi-turn interaction and assesses along four dimensions: task utility (end-to-end accuracy and checkpoint pass rate), ambiguity detection capability (detection accuracy and F1 score), interaction strategy (clarification accuracy and advancement rate), and cost efficiency (number of questions asked, number of tool calls, token consumption).

In addition, the team designed a User Simulator. User simulators have a long history in dialogue systems research, from early rule-based simulators (such as the agenda-based simulator for task-oriented dialogue) to today's LLM-based neural simulators, whose core goal has always been to achieve repeatable, controllable interactive evaluation without requiring real user participation. A key metric for measuring simulator quality is Ecological Validity, i.e., the degree to which simulated behavior resembles real user behavior. The biggest flaw of traditional simulators is the "god's-eye view"—the simulator knows all the answers in advance, causing information disclosure in the dialogue to be overly idealized and unable to truly reflect users' information-provision behavior under uncertainty. DiscoBench's user simulator adopts an "on-demand progressive information disclosure" strategy—giving "distinguishing clues step by step" in response to the model's questions, rather than disclosing all information at once. This precisely simulates real users' "provide information on demand" behavior, effectively avoiding the evaluation bias caused by the "god's-eye view" in traditional simulators, such that the quality of the model's questions directly affects the amount of information it can obtain. This design substantially reduces testing costs without human participation while still possessing the ecological validity of real interaction.

The test sets up two prompt modes: neutral mode provides no hint that ambiguity exists, testing the model's spontaneous perception capability and more closely matching real usage; guided mode explicitly reminds the model to watch for potential ambiguity, testing its performance ceiling. By comparing the two, one can see whether the bottleneck is "not realizing clarification is needed" or "knowing clarification is needed but doing it poorly."

The Real Performance of Mainstream LLMs

The experimental results are far from ideal. In neutral mode, the best-performing Doubao Seed 2.0 Pro achieved an end-to-end accuracy of only 43.1%, Gemini 3.1 Pro Preview reached 40.8%, most models fell below 40%, and the weaker Qwen 3.6 Max scored just 12.3%.

A noteworthy phenomenon: many models' intermediate checkpoint pass rates were far higher than their final accuracy. For instance, Claude Opus 4.7's checkpoint pass rate reached 57%, yet its final accuracy was only 39.8%. This is precisely a direct manifestation of the aforementioned "error cascading" effect—in an n-step reasoning chain, mishandling at each ambiguity point gets exponentially amplified by subsequent steps, and local success cannot guarantee global correctness. Consistently stable ambiguity recognition capability throughout is the true key.

In guided mode, average end-to-end accuracy improved from 28.6% to 33.7%, and the ambiguity detection F1 score improved most significantly, rising from 45.3% to 64.9%. But the improvement was mainly concentrated in reducing missed detections, and the improvement in final end-to-end performance was limited—Claude Opus 4.7 even had a higher checkpoint pass rate in guided mode yet a slight decline in final accuracy. This indicates that prompt engineering can only partially solve the problem; the fundamental bottleneck lies in the model's insufficient ambiguity clarification capability.

Ambiguity Perception and Effective Questioning: Two Independent Capabilities

The paper reveals an important conclusion: "knowing when to ask" and "knowing how to ask effective questions" are two completely independent capabilities.

From an information-theoretic perspective, the optimal clarification question can be formalized as a decision problem of "maximizing conditional entropy reduction"—that is, selecting the question that can most greatly eliminate the system's uncertainty about the user's true intent. This line of thinking is highly isomorphic to the Information Gain calculation in decision trees: a good clarification question is equivalent to selecting the node with the strongest classification ability for splitting in a decision tree; the more a question can reduce the system's uncertainty about user intent, the higher its information gain. Specifically, if the user intent space has n possible interpretations, the optimal question should split these n interpretations as evenly as possible into two subsets, so that no matter how the user answers, the system's uncertainty is halved—this is precisely why binary clarification questions (yes/no type) are theoretically the most efficient.

It's worth adding that how to automatically generate high-quality clarification questions is itself one of the core challenges in dialogue systems research. Early methods relied on templates and slot filling, applicable only to restricted domains. In recent years, clarification question generation based on Retrieval-Augmented Generation (RAG) has become mainstream: first retrieve candidate documents relevant to the current query, then generate targeted questions based on the ambiguity points revealed by the documents. Another technical path is confusion-matrix-based question selection—enumerating in advance all possible interpretations of the user intent space, transforming clarification question generation into a selection optimization problem of "which question can best distinguish these interpretations," directly corresponding to the theoretical framework of information gain maximization.

Supplementing this understanding from a game-theoretic perspective: a search agent's clarification decision is essentially a dynamic game of incomplete information, where the agent is the "questioner" and the user is the "information holder." The Bayesian updating mechanism is the optimal framework for understanding the entire clarification process—after each question, the agent updates its posterior probability distribution over the user's intent based on the user's answer, until uncertainty drops to an acceptable threshold. This perspective also explains why the "search-then-ask" strategy achieves higher clarification quality: the search action simultaneously updates the agent's beliefs about the state of the external world, and the joint updating of these two beliefs significantly increases the information gain of subsequent clarification questions.

The experimental data clearly corroborates this framework: Qwen 3.6 Max in neutral mode had an ambiguity detection F1 of only 16%, asking on average just 0.07 questions per task, almost never proactively asking; but once it did ask, its clarification accuracy reached as high as 94.7%—the questions it asked were of very high quality (extremely high information gain); it simply didn't realize it needed to ask. Conversely, MiniMax M2.7 had a very high questioning frequency (0.6 questions per task), but its clarification advancement rate was only 60.7%, asking many ineffective questions (low information gain questions). An excellent ambiguity clarification system must possess both.

Another counterintuitive conclusion is: more tool calls do not necessarily bring better performance. Claude Opus 4.7 had a very high tool-calling frequency, yet its accuracy was lower than that of Gemini and Doubao, which called fewer tools. Many models had already discovered multiple candidate entities yet repeatedly searched without stopping to ask, ultimately going further and further down the wrong path. This phenomenon is especially noteworthy in RL-trained models: if excessive positive reinforcement is given to tool-invocation behavior during training, the model may learn a strategy preference of "calling more tools is always safer than stopping to ask," even though this strategy is counterproductive in ambiguous scenarios. This suggests that when designing training signals, the industry needs to give explicit negative reinforcement to "ineffective tool calls" (i.e., calls that fail to reduce uncertainty), rather than only rewarding or penalizing the correctness of the final answer.

Behavioral Pattern Analysis and Ablation Validation

The team classified the models' behavior at ambiguity points into four categories: direct guess, guess after multiple searches, direct ask, and search-then-ask. The results show that "search-then-ask" had the highest success rate, with an average pass rate of 93.4%, far higher than the 56.5% of direct guessing. More interestingly, "guess after multiple searches" (51.5%) was even lower than "direct guess"—the model had actually realized there was ambiguity, which is why it searched repeatedly, but was unwilling or didn't know to ask the user, and instead wasted more resources. This is precisely the primary failure mode of current search agents.

The reason the "search-then-ask" strategy is most effective is that the search action itself has the byproduct value of "ambiguity diagnosis": after obtaining more context through searching, the model can more precisely identify the specific nature of the ambiguity, thereby posing more targeted clarification questions, rather than broadly asking the user "what do you mean?" This finding has direct implications for industry system design: ambiguity clarification should not be divorced from the retrieval process but should be tightly embedded within the reasoning loop. In systems engineering terms, this means the position of the "clarification trigger" should shift from the starting node of the process (Pre-search clarification) to intermediate nodes (Mid-search clarification), deeply coupled with the retrieval loop rather than existing as an independent front-end module.

Ablation experiments further validated the reasonableness of the benchmark. After removing the search tool, Doubao Seed 2.0 Pro's accuracy plummeted from 43.1% to 2.4%, proving that the task cannot be solved by parametric knowledge and must rely on external retrieval. And after removing ambiguity, all models' accuracy improved by 26.8 to 40.2 percentage points, with Gemini even reaching 81%. This clearly shows: ambiguity is one of the biggest performance bottlenecks for current deep search agents.

Implications for Search Product Development

This research provides a clear direction for search product development. In the past, everyone was focused on improving reasoning and retrieval capabilities; now it seems that ambiguity perception and proactive clarification must be built as core modules. The clarification mechanism cannot rely solely on prompt guidance; it requires dedicated modules to judge when to ask and how to pose effective questions, and to connect retrieval uncertainty with user interaction.

The industry already has several clarification-triggering strategies worth referencing: confidence-threshold-based triggering mechanisms are widely applied in intelligent customer service, triggering follow-up questions when the model's classification confidence about user intent falls below a preset threshold; hybrid output strategies (such as Perplexity's "related questions" recommendations) are essentially a form of implicit clarification, letting users choose autonomously by presenting multiple possible directions; Implicit Clarification—i.e., embedding assumption statements in the answer, such as "Assuming you mean mainland China box office..."—provides a compromise between user experience and clarification efficiency. From a product engineering perspective, these three strategies each have advantages in different scenarios: confidence-threshold triggering suits high-risk vertical scenarios where wrong answers carry high costs (such as medical or legal consultation); implicit clarification suits low-risk general search scenarios with high user tolerance; hybrid output strategies retain the effect of explicit clarification while accommodating user experience. DiscoBench's research results show that the effectiveness of these strategies is highly dependent on the ambiguity perception precision of the underlying model—if ambiguity perception capability is insufficient, even the most elegant triggering strategy will struggle to be effective.

From a user experience standpoint, many inaccurate search results are not because the model couldn't find them, but because it misunderstood from the start. Asking the user a question at the right node can both improve accuracy and reduce the hassle of repeatedly revising the query. But there's a balance here—you can't ask about everything, or users will get annoyed. The ideal state is that the model can judge for itself the severity of the ambiguity: if the cost of choosing wrong is low, it can directly provide multiple results; if the cost is high, it must proactively clarify. This judgment is essentially an expected loss minimization problem: if the user experience cost brought by clarification is lower than the answer quality loss caused by ambiguity, then proactive clarification is warranted; otherwise, the most likely answer can be given directly with additional explanation. From the perspective of Bayesian decision theory, the optimal decision depends on the product of two parameters: the probability that ambiguity leads to a wrong answer, and the ratio of the utility loss of a wrong answer relative to the utility loss of interrupting the user. Only when this product exceeds a threshold is proactive clarification the rational choice—this provides a formal basis for building adaptive clarification-triggering modules in the future.

It's worth mentioning that DiscoBench is primarily oriented toward Chinese scenarios, filling the gap where previous ambiguity datasets were mostly in English. Chinese naturally has higher ambiguity density in its linguistic structure: broad pronoun reference scope, a high proportion of polysemous characters, extensive reliance on subject-omitting expression habits, and rich pragmatic conventions dependent on shared knowledge backgrounds—these characteristics make pragmatic ambiguity in Chinese queries more prevalent than in English, and also make a dedicated Chinese benchmark of higher reference value to domestic LLM teams. The team has explicitly stated that the code and data will be made public soon.

Future improvement directions include: early prediction of ambiguity, more precise uncertainty modeling, personalized clarification strategies, and multimodal clarification. Ultimately, this research prompts us to re-examine the essence of search—it is not simple information matching, but a process in which the user and the system collaborate to solve information needs. When a system faces uncertainty and proactively communicates, rather than presumptuously giving a wrong answer, this is precisely a key step in AI's transition from "passive response" to "proactive collaboration."

Key Takeaways

Key Takeaways