120 Tool Selection Questions Tested: ChatGPT, Perplexity, and Gemini Recommendations Overlap Only 14% of the Time

Hand a purchasing decision to an AI agent, and the answer depends heavily on which engine you hand it to. A Reddit user ran a controlled experiment using 120 real-world tool selection questions, and the results were more fragmented than most people would expect: across four engines, 42% of questions produced recommendation lists with zero products in common.

Experiment Design: 120 Questions Across Six Categories

The author collected 120 questions that people actually type when evaluating tools — things like "best X API," "cheapest Y," or "alternatives to Z." These covered six categories: lead enrichment, Clay alternatives, agentic MCP tools, AI visibility tracking, SEO data APIs, and email verification.

Each question was sent to four "engine configurations": ChatGPT with web search enabled, ChatGPT with web search disabled, Perplexity, and Gemini. That's 480 total calls, yielding 472 valid responses. The author then identified which product names appeared in each response using string matching against a manually curated brand list.

The author openly acknowledges the roughness of this approach: it only does name matching, so any product not on the brand list is invisible; and nobody verified whether the recommendations were actually good. The experiment measures consistency, not accuracy.

Lead Enrichment refers to using third-party data sources to fill in and enrich existing prospect records — for example, adding company size, contact title, email address, or tech stack information. These tools are widely used in B2B sales automation, with typical products including Apollo, Clearbit, and ZoomInfo. MCP (Model Context Protocol) is an open protocol introduced by Anthropic in late 2024, designed to standardize how AI models interact with external tools and data sources, allowing agents to call file systems, databases, and APIs through a unified interface. Because the protocol was released recently, the market is flooded with new tools but lacks a settled landscape — which explains why the MCP category showed the highest engine disagreement in this experiment. Coverage of this emerging ecosystem varies enormously across models' training data and real-time retrieval results.

Key Finding: 42% Zero Overlap, Median Overlap Rate Just 0.14

Among the 112 questions where all four engines responded, 47 (42%) had zero product overlap across all four recommendation lists. The median Jaccard similarity — intersection divided by union — was just 0.14. In other words, the four engines were overwhelmingly talking past each other.

The dispersion is even more striking: 37% of all mentioned products were cited by only one engine, with the other three never mentioning them at all. That means more than a third of all recommendations were "singletons" — if you happened to use a different engine, you'd never see them.

The most fragmented category was "agentic MCP tools": 12 of 19 questions had zero overlap. This emerging space lacks stable consensus, and the differences in models' knowledge and retrieval results are amplified further.

The overlap metric used in this experiment is the Jaccard similarity coefficient (intersection size divided by union size), a standard metric in information retrieval and set comparison, ranging from 0 to 1. A score of 1 means all four engines recommended identical products; 0 means no overlap at all. A median of 0.14 means that across all products collectively listed by the four engines, only about 14% were mentioned by all of them — the remaining 86% were unique to certain engines. This metric is useful because it accounts for differences in list length: even if one engine returns a longer list, the Jaccard coefficient still objectively reflects the degree of overlap.

Same Engine, Different Results With or Without Search

One easily overlooked detail: even within the same ChatGPT model, turning web search on versus off produced lists that overlapped only 29% of the time. This shows that the divergence isn't just about differences between vendors' models — it also comes from whether real-time retrieval is involved. Enabling search pulls answers toward whatever is trending on the web at that moment, while disabling it relies more on what the model learned during training.

The author also noted something he found amusing: when asked for "alternatives to Clay," 95% of responses still included Clay itself. This exposes a common failure mode in how models handle "exclusion queries" — they surface the excluded item as a highly relevant result anyway.

This touches on two distinct knowledge-sourcing mechanisms in large language models. Parametric memory refers to knowledge compressed and encoded into the neural network's weights during pretraining. It requires no internet access at inference time, but is limited by a knowledge cutoff date and tends to systematically underrepresent products with low training data exposure. Retrieval-Augmented Generation (RAG) fetches external documents or web pages in real time during inference and injects them into the prompt — web search is essentially a form of RAG. RAG can access current information, but results are influenced by search engine rankings and SEO optimization, which tend to favor well-known products. These two mechanisms each introduce their own distinct biases, which explains why the same model with different retrieval settings produced 71% non-overlapping recommendations in this experiment.

Limitations: A Single Sample Isn't Enough to Draw Conclusions

The author showed admirable restraint about his own experiment. He explicitly noted that this was a one-time sample conducted on a single day with no repeated trials — and that repetition is exactly what he plans to add next week before claiming the numbers are stable.

The problem with single-sample testing is that LLM outputs are inherently stochastic; the same question asked multiple times can yield different answers. So it's currently impossible to determine how much of that 42% zero-overlap rate reflects systematic differences between engines versus simple sampling noise. This is a general caveat worth keeping in mind for this type of comparative experiment.

Takeaway: Don't Hand "Which Tool Should I Buy" Directly to a Model

The author's own work happens to fall within one of the six categories tested. For person and email lookups, he uses a pay-per-call aggregation layer called Treg — one API key, one balance, with automatic vendor selection based on price and hit rate, eliminating the need to register with each provider separately. That tool appeared exactly zero times across all 472 responses.

He wasn't surprised. For a niche, newer product, being overlooked is expected. But that's precisely why he no longer asks models "which tool should I buy" — model recommendations carry significant cross-engine bias and are systematically skewed toward high-visibility products, with blind spots for emerging or niche solutions.

For teams relying on AI to assist with procurement or vendor selection, this experiment offers a practical reminder: treat model recommendations as a starting point, not a conclusion — especially in emerging technology categories. It's worth cross-referencing multiple engines and preserving a step for independent verification. Expecting a single AI to deliver the "optimal tool" may just mean getting that engine's preferences.