Dissecting Web Search in Conversational AI: Which Is More Reliable — ChatGPT, Claude, or Grok?

Conversational large language model (LLM) agents are increasingly relying on web search to access real-time information. Yet the complete pipeline — from deciding when to search and how to construct queries, to ultimately generating a response — has never been systematically studied. A new paper published on arXiv (arXiv:2609.19244) offers the first comprehensive end-to-end analysis of this process, covering four major conversational platforms: ChatGPT, Claude, Grok, and DeepSeek.

What makes this research valuable is that it goes beyond asking whether a model answers correctly — it digs into why the model answers the way it does. Specifically, it examines agent behavior across four dimensions: retrieval decision-making, query strategy, result sourcing, and answer generation. This provides empirical grounding for understanding the reliability limits of today's AI assistants, and points toward better design principles for future AI agents and search tool integration.

Research Methodology: Real-World Interaction Meets Controlled Experimentation

The research team employed a two-pronged methodology. On one hand, they conducted in vivo observations by collecting real-world interaction data between users and these platforms. On the other hand, they ran in vitro controlled experiments using the same underlying models accessed through each platform's official API.

This combination is significant: real-world interactions capture search behaviors triggered naturally by users, while controlled experiments eliminate confounding variables and enable meaningful cross-platform and cross-model comparisons. The choice of four platforms is also representative — ChatGPT and Claude represent leading Western AI providers, Grok is deeply tied to real-time social platform data, and DeepSeek is a prominent open-source model that has attracted considerable recent attention.

The study centers on four core questions: How well do agents decide when to invoke web search? What strategies do they use to construct queries? Do the returned search results exhibit domain bias? And how do agents convert search results into well-attributed answers?

Key Finding 1: More Searching Doesn't Mean Better Answers

One counterintuitive finding is that the frequency of web search invocation varies significantly across platforms and models — and more frequent search does not necessarily lead to higher-quality responses.

This challenges the naive assumption that "more retrieval equals more reliability." It suggests that the quality of the search decision matters far more than the quantity: an agent that knows when to trigger retrieval — for the right question, at the right moment — may outperform one that searches reflexively. Over-retrieval wastes compute and latency, and can introduce noisy information that actually dilutes response accuracy.

For product designers, this points to a clear optimization target: rather than maximizing search call frequency, the focus should be on refining the model's judgment about when to search. Knowing when to rely on parametric knowledge versus external retrieval is the key decision variable that shapes user experience.

Key Finding 2: Query Strategies and Domain Bias in Search Results

The research also reveals that different conversational agents employ distinct and complex query strategies. Faced with the same user question, the models across platforms each have their own approach to decomposing, rephrasing, and combining search keywords — which directly shapes what information they are able to retrieve.

Even more concerning is the issue of domain bias. The study found that the search engines underlying specific platforms tend to return results from their "preferred domains." This means that different AI assistants answering the same question may be influenced by their respective retrieval backends' information filtering right from the source.

This preference may stem from commercial partnerships, data compliance requirements, or content quality assessments — but for users, it constitutes a form of hidden information bias. When you ask the same question to different AI assistants and get different answers, the divergence may not originate from the models themselves, but from the fact that they are each seeing a different slice of the world.

The "domain preference" of search engines is closely tied to their ranking algorithms, crawling strategies, and commercial partnerships at the technical level. Take Microsoft Bing as an example: early versions of ChatGPT's search functionality relied on Bing's backend, meaning Bing's domain authority scoring system directly influenced what content the model could "see." At the same time, different platforms may have content licensing agreements that restrict or prioritize results from specific media outlets. For Grok, its deep integration with X (formerly Twitter) real-time data creates a natural tilt toward social media content. This structural preference is fundamentally similar to the "filter bubble" phenomenon in traditional search — except in the AI agent context, users have even less awareness and control over the mechanism. Traditional search users can still try different keywords or switch engines to break out of information silos; with conversational AI, the retrieval process is entirely opaque to the user.

Key Finding 3: Missing Citations and the Credibility Gap

In the answer generation phase, the study found that while conversational agents' responses are broadly grounded in search results, a portion of the claims they make rely on search results that are never cited.

In other words, the model uses certain retrieved information when generating its response but fails to attribute that information in the answer. This directly raises concerns around attribution and reliability. Users have no way to trace the source of a specific claim, making it difficult to assess its credibility or perform fact-checking.

In an era where information integrity is increasingly critical, this issue is particularly sensitive. A responsible AI assistant should not only be accurate but also be accountable — enabling users to follow the trail and verify information. The widespread absence of citations reveals a clear shortfall in transparency among current conversational AI systems.

Attribution has a specific meaning in the AI field: it requires the model not only to provide an answer, but to explicitly identify which specific source that answer came from. This is analogous to citation norms in academic writing, but significantly harder to implement in conversational AI — the model must track the provenance of information through three steps: retrieval, integration, and generation. While current mainstream RAG (Retrieval-Augmented Generation) architectures theoretically support citation tracking, in practice models often blend and paraphrase content from multiple sources, causing the correspondence between specific claims and their original sources to become blurry or broken. Additionally, some platforms, prioritizing interface simplicity, do not present full citation information to users. This "use without disclosure" behavior can meaningfully mislead user decision-making in high-stakes domains such as healthcare, legal matters, and finance.

Implications for Future AI Agent Design

The study's overall findings point to several improvement directions for AI agents and search tools optimized for conversational retrieval.

First, retrieval decision mechanisms need to be more refined — agents should learn to "act only when the time is right," rather than blindly maximizing search frequency. Second, query construction strategies should be more transparent and evaluable, to prevent poor strategies from leading to low-quality retrieved content. Third, domain bias in search backends must be acknowledged; an ideal agent should strive to provide diverse, unbiased sources. Fourth, answer generation must enforce stronger citation standards, ensuring that every claim derived from external retrieval is traceable.

As the first study to systematically characterize the full lifecycle of web search by conversational LLM agents, this work lays important methodological and empirical groundwork for future research. For everyday users, it is also a reminder: when an AI assistant presents a "factual" answer, a degree of healthy skepticism — and a habit of actively verifying the sources of key claims — remains the wisest approach.