HybridDeepResearch: The First Hybrid Deep Research Benchmark Reveals Cross-Modal Integration Bottlenecks in AI Agents

The Real-World Dilemma Facing Deep Research Agents

In recent years, autonomous agents have made significant strides in the field of "deep research." Autonomous agents are AI systems capable of perceiving their environment, making decisions, and independently executing actions to achieve goals—unlike traditional single-turn Q&A models, these agents possess multi-step reasoning, tool-calling, and environment interaction capabilities. In "deep research" scenarios, they typically employ ReAct (Reasoning + Acting) or similar thought-action loop frameworks, autonomously deciding what to search next, what to query, and how to synthesize the information they've gathered. Products like OpenAI's Deep Research and Google's Gemini Deep Research are representative examples of this direction. They can iteratively browse the open web, synthesize information from different web pages, and ultimately formulate answers to complex questions. However, real-world problem-solving is never confined to a single environment.

A recent paper from Snowflake AI Research, Benchmarking Hybrid Deep Research Across Database Querying and Web Search (arXiv:2609.09410), identifies a critical issue: complex analytical tasks inherently require agents to weave together evidence from ambiguous, unstructured text (such as the open web) and highly precise, structured data (such as relational databases).

It's important to understand the fundamental differences between these two types of data: structured data is organized according to a predefined schema, with tabular data in relational databases being the classic example—it can be precisely queried via SQL, and the results are deterministic and reproducible. Unstructured data, on the other hand, includes web page text, news articles, social media posts, and so on—these have no fixed format, uneven information density, and often contain ambiguity, redundancy, and noise. SQL queries return exactly matching records, while web searches return approximate results ranked by relevance—this fundamental difference is precisely what makes hybrid reasoning tasks so challenging.

In reality, analysts do this kind of work every day—they query company databases for precise transaction records while also searching the web for background information, news, and industry trends, then integrate both types of information into a complete conclusion. Yet existing evaluation benchmarks assess these two modalities in isolation, failing to capture the most critical "handoff" between them.

What Is an Agent's "Handoff" Capability?

The "handoff" capability emphasized by the research team refers to the ability to preserve constraints when moving evidence between different systems. This is a challenge that has been severely underestimated by previous research.

Consider an example: suppose you need to answer "What is the social media sentiment about our company's top three products by revenue last year?" This task requires the agent to first query a database via SQL to identify the "top three products by revenue" (structured data), then carry those precise constraints to search for relevant reviews on the web (unstructured data).

During this process, the agent must accurately pass the product names retrieved from the database query to the web search step—without losing, altering, or blurring these constraints. This problem is analogous to the concept of a "data pipeline" in software engineering—in multi-step reasoning, each step's output serves as the next step's input, and information loss at any point propagates and even amplifies along the chain, similar to the noise accumulation effect in signal processing. In agent systems, typical manifestations of information loss include: the model losing precise numerical constraints when converting SQL query results into search keywords (e.g., "top three by revenue" becomes "high revenue"), entity names subtly morphing during transfer (e.g., abbreviations being expanded or spelling changes), and qualifying conditions like time ranges being forgotten during cross-step transfer. The root cause of these issues lies in the fact that current large language models rely on natural language as an intermediate representation, and natural language is inherently ambiguous—unlike programming languages, which can precisely preserve semantics. Once information loss occurs during the "handoff" process, the entire answer veers off course. This is precisely the core challenge facing current AI agent systems.

Detailed Design of the HybridDeepResearch Benchmark

To systematically evaluate this capability, the research team proposed HybridDeepResearch—to their knowledge, the first deep research benchmark that simultaneously requires web search and SQL querying to form complete, verifiable answers.

Data Scale and Construction Method

The benchmark contains 380 tool-dependent tasks built upon the LiveSQLBench-Base-Lite database and publicly available web corpora. LiveSQLBench is a benchmark test suite specifically designed to evaluate LLMs' SQL generation capabilities, using real-world database scenarios to test models' ability to translate natural language questions into correct SQL queries. HybridDeepResearch innovatively extends this foundation by coupling SQL query tasks with web search tasks.

All tasks underwent dual verification through automated checks and human review—automated checks ensure that each task genuinely requires both tools to complete (rather than being answerable with just one), while human review validates answer correctness and task description clarity. The scale of 380 tasks is moderate for a research benchmark, but since each task requires cross-modal reasoning, its evaluation value far exceeds simply scaling up data volume.

Three Core Reasoning Modes

The benchmark covers three distinct reasoning modes, which form the essence of its design:

• SQL2S (SQL-to-Search): First perform an SQL query, then take the results to search the web (directional reasoning from structured to unstructured)
• S2SQL (Search-to-SQL): First perform a web search, then use the results for an SQL query (directional reasoning from unstructured to structured)
• Parallel: Gather evidence from both information sources in parallel, then perform cross-validation at the end

Distinguishing these three modes is crucial because they represent different levels of difficulty and different types of cross-modal integration challenges. SQL2S requires the model to effectively transform precise database results into web search strategies; S2SQL requires the model to extract sufficiently precise conditions from fuzzy web information to construct SQL queries; and the Parallel mode tests the model's ability to simultaneously manage two independent information streams and ultimately perform cross-validation.

Even Top Models Barely Break 50%: An In-Depth Analysis of Evaluation Results

The research team conducted extensive evaluations across proprietary models and open-weight models using multiple agentic scaffolds. An agentic scaffold is a systematic architecture built around a large language model that defines how the model perceives its environment, plans actions, calls tools, and integrates results. These frameworks typically include several core components: a task planner (decomposing complex problems into subtasks), tool-calling interfaces (connecting to external systems like search engines and databases), a memory module (maintaining conversation history and intermediate results), and a reflection mechanism (evaluating current progress and adjusting strategies). Different framework designs significantly impact agent performance—whether backtracking is supported, whether plans can be dynamically adjusted, and the granularity of tool calls are all critical design choices.

The evaluation results are thought-provoking. Even the most advanced current models—such as GLM-5.2, Claude-Sonnet-4.6, and GPT-5—achieved only about 50-54% Pass@8 on the hard subset.

Pass@k is an evaluation metric originating from the code generation domain, first widely used by OpenAI in the Codex paper. It represents the probability of at least one success across k independent attempts. A Pass@8 of approximately 50-54% means that even when giving the model 8 independent chances to generate an answer (each potentially taking different reasoning paths and tool-calling strategies), only about half of the difficult tasks can be solved at least once. Since this metric is more lenient than Pass@1 (single-attempt success rate), a Pass@8 of 50-54% actually implies that the single-attempt success rate is far lower—indicating that the problem isn't merely one of sampling luck, but rather a systematic insufficiency in model capabilities.

This number clearly demonstrates that while individual capabilities (whether pure web search or pure SQL querying) are already quite mature, effectively combining the two remains a formidable barrier.

Directional Reasoning Is Harder Than Parallel Cross-Validation

A particularly noteworthy finding is that directional reasoning is significantly more difficult than parallel intersection.

This means that tasks like SQL2S and S2SQL—which require strict sequencing and demand that precise output from one step serve as input for the next—pose a greater challenge for agents. In contrast, tasks that gather evidence in parallel before computing an intersection are relatively easier.

The underlying logic isn't hard to grasp: in directional reasoning, constraints from the previous step must be transmitted completely and intact to the next step, and any information loss amplifies along the reasoning chain—what complex systems theory calls "cascading failure," where a small deviation in one link can cause severe errors downstream. In parallel mode, the two information sources are relatively independent, and only an intersection verification is needed at the end, leaving more room for error tolerance. This finding also aligns with research in human cognitive science: sequentially dependent reasoning tasks tend to be more error-prone than tasks that can be processed in parallel, because working memory must precisely maintain contextual information between steps.

Practical Implications for the AI Agent Industry

This research reveals a fundamental shortcoming in current agent systems: crossing the boundary between structured and unstructured information spaces without losing constraints remains a major unsolved challenge for agent systems.

For teams building enterprise-grade AI applications, this finding carries significant practical implications. Many enterprise scenarios (business intelligence analysis, compliance review, market research, etc.) are inherently hybrid tasks requiring both internal database queries and external information retrieval. Take compliance review as an example: reviewers need to identify anomalous transaction patterns from internal transaction databases (structured queries), then confirm in regulatory announcements, news reports, and industry guidelines whether these patterns constitute violations (unstructured retrieval), and finally synthesize both into a review opinion. If agents frequently err during the "handoff" step, their reliability in these high-value scenarios becomes questionable.

This also points the way for future research directions: designing more robust constraint-passing mechanisms and maintaining contextual consistency across multi-tool calls will become important research topics for agent systems. Potential solutions may include: introducing formalized intermediate representation languages to replace natural language for cross-step information transfer, designing specialized constraint-verification modules to check information integrity at each "handoff," and using reinforcement learning to train agents to better maintain constraint consistency during multi-step reasoning.

Open Resources and Community Participation

Commendably, the research team has embraced the spirit of openness by making their code and dataset fully public:

• GitHub: https://github.com/Snowflake-AI-Research/HybridDeepResearch
• Hugging Face: https://huggingface.co/datasets/Snowflake/HybridDeepResearch

This provides the community with valuable infrastructure for further research on hybrid deep research tasks and improving agents' cross-modal integration capabilities. Open benchmarks play an important catalytic role in AI research—just as ImageNet drove the rapid advancement of computer vision and SQuAD pushed forward reading comprehension capabilities, HybridDeepResearch has the potential to become a key benchmark driving progress in hybrid reasoning agents. As more and more researchers evaluate and optimize on this benchmark, we have every reason to expect the next generation of agents to truly handle the complex real-world tasks that demand multi-source information integration.