AI Web Search Showdown: Claude, Codex, Kimi K3, and Qwen Tested 120 Times

The Core Question Has Changed

Let me start with a counterintuitive conclusion: in this 120-run evaluation, the only configuration that got everything right was cached mode search.

Four months ago, most people discussing AI search were still arguing about "which search engine does it use under the hood." That question is now outdated. None of these four — Claude, Codex, Kimi K3, and Qwen — discloses its search engine in official documentation. The question actually worth asking is: where does search run, and who invokes it?

Comparison test of four AI search tools

OpenAI makes its in-house index the default for Codex; Kimi has spun search out as a standalone API. This shift isn't just on paper — the test results were just as surprising.

Architectural Differences in How Each Search Runs

Before comparing anything, let's look at the mechanical differences.

	Claude	Codex / OpenAI	Kimi	Qwen
Where search runs	Anthropic server-side	OpenAI-hosted tool	Your code calls the API and returns results	Alibaba Cloud DashScope; local Qwen has no search
Freshness control	None	Cached by default, `--search` switches to real-time	Pro search supports time windows	7–365 day window
Cost per search	$0.01	$0.01	$0.002–0.003	Varies by region
Search engine	Undisclosed	Undisclosed	Undisclosed	Undisclosed

Both Claude and OpenAI perform search on their own servers before handing results to the model. Claude returns results encrypted — you must pass them back verbatim to continue the conversation.

Kimi is in a transition period. The built-in $web_search tool will be retired on October 20, 2026, replaced by a standalone REST API such as /v1/tools/search. The new API lets you decide when to search and how many times, costs one-third to one-fifth of the alternatives, and only bills when results are actually returned.

Qwen models have no search capability on their own. Alibaba Cloud's API provides search, but locally run Qwen requires you to manually wire in an external search tool.

Two Easily Overlooked Details in the Docs

Claude Code's WebSearch returns only titles and URLs, not page content. Reading pages is WebFetch's job, and WebFetch first calls another model to summarize. The documentation itself calls this design "lossy by design."

Qwen search on Alibaba Cloud silently skips search when exceeding 15 requests per second. No error — the search simply doesn't execute. The FAQ also notes that English-language queries often fail to trigger search, recommending Chinese queries or forced enabling. These two notes only appear in the Chinese documentation.

I hit the "lossy" pitfall firsthand while writing this article: I asked Claude Code's WebFetch to read the Qwen docs, and it reported that the 15 RPS limit and English FAQ weren't there. Switching to grep on the raw HTML (~460KB), I found both. Once a summarization layer sits in between, "the docs don't say that" may just mean "the summary dropped it."

The "server-side search" and "client-side search" that appear repeatedly in this article don't refer to physical location, but rather to who holds control over the search pipeline. Server-side search (Claude, Codex) means: after your request goes out, the vendor performs retrieval, crawling, and summarization on its own infrastructure, then returns the processed results along with the model's answer. You can't see the raw search results, nor can you preserve search state between calls. Client-side search (Kimi's new REST API, local Qwen) is the opposite: search is a standalone HTTP endpoint, and your code decides when to call it, how many times, and what results to feed back to the model. This distinction has three practical consequences. First, observability: in client mode you can see every query term, hit URL, and raw snippet, making it easy to diagnose whether "search didn't find it" or "the model didn't understand it." Second, controllability: you can insert your own re-ranking, deduplication, or caching layers. Third, the cost-latency tradeoff: an extra round trip adds latency, but you only pay when results are returned. The emphasis on "where search runs" replacing "which engine" essentially reflects that once control shifts from vendors to developers, the evaluation dimension shifts from "engine quality" to "overall pipeline quality."

The RESTful search APIs mentioned here run into rate limiting and silent degradation issues in practice. Cloud services typically use requests per second (RPS or QPS) to cap call frequency, usually returning HTTP 429 with a Retry-After header so clients can back off and retry. Alibaba Cloud's Qwen search silently skips search after 15 RPS — no error, no "skipped" marker, just turns that call into an ordinary request with no retrieval results. This design is extremely unfriendly to callers, because the pipeline can't distinguish "searched but got nothing" from "didn't search at all," leaving you to diagnose by comparing latency and billing logs. Another common phenomenon is query language bias: retrieval indexes and query-rewriting modules are often trained or sourced primarily in one language, so recall plummets when you ask in English about a Chinese-language site, and improves when you query in the site's native language. The fact that these two points only appear in Chinese documentation also suggests that vendors provide unequal transparency to users of different languages — something every developer building their own search pipeline should watch for.

Test Methodology

Five configurations, 8 questions, each run 3 times — 120 graded answers total.

Claude: Claude Code with WebSearch/WebFetch
OpenAI real-time: Responses API web_search
OpenAI cached: Same, plus external_web_access: false
Kimi K3: Connected to the Kimi search API
Local Qwen: Running on Ollama with the same search function

The 8 questions covered post-training-cutoff news, date traps, Japanese-language sources, freshness, documentation details, number extraction, and "does it exist" bait. Questions q3 and q4 were about my own book and an article I published that same day — I chose them because I knew the answers, but this does introduce bias, and I'd rather disclose it upfront. Both were in Japanese, which also conveniently tests each configuration's coverage of Japanese-language sources.

To understand this evaluation's conclusions, you need to grasp a few evaluation terms. Post-cutoff questions refer to events whose answers came after the model's training data cutoff; the model can't answer from memory and must rely on retrieval, making them the cleanest type of question for testing the search pipeline. But such questions have inherent flaws: the model may "happen to remember" some related background and partially guess right, so they're usually paired with date trap questions (where the answer is an easily confused date) and existence questions (which bait the model into fabricating an answer about something that doesn't exist) for cross-validation. Three repeat runs serve to distinguish "consistently correct" from "occasionally lucky" — if the same configuration gives different answers across three runs, the pipeline has randomness and a single score can't be trusted. Median latency rather than average latency is used to exclude outliers from skewing the mean. Note that latency across configurations isn't directly comparable: Claude Code includes the agent's multi-turn decision loop, while direct API calls are a single round trip — these two have completely different time structures, and listing them side by side is only to show magnitude, not performance ranking.

Results: Cached Mode Gets Everything Right

Configuration	Correct (/24)	Partially correct	Wrong or no answer	Median latency	Avg. searches
Claude	21	1	2	22.0s	3.5
OpenAI real-time	22	1	1	5.7s	2.3
OpenAI cached	24	0	0	6.2s	2.7
Kimi K3	21	0	3	25.9s	4.0
Local Qwen	9	2	13	31.1s	1.7

Finding 1: Cache Isn't Stale

Cached search got all 24 questions right. The article in q4 went live around 5 PM that day, and I started testing about 3 hours later. Cached search returned the correct title in all 3 runs. At least for my site, OpenAI's maintained index refreshed within hours.

Cache sometimes also had better source quality. Q6 asked about Kimi's $web_search retirement date; cached mode cited Kimi's official documentation, while real-time search gave the same answer but once cited an unofficial aggregator site. The broader the coverage, the higher the chance of pulling in non-official pages.

Finding 2: Same Search Results, Different Answers

Kimi K3 and local Qwen got the exact same search API, invoked in exactly the same way. The scores were 21 versus 9 — more than double the difference.

Qwen's errors fell into several patterns:

It firmly believed it's currently 2024, and that belief overrode the search results. Its answer to q4: "As of now (May 2024), an article dated October 14, 2026 doesn't exist — that's a future date."
The sources it cited contradicted the answers it gave. For q5, all three runs cited the May 21 release candidate announcement but answered with the July 28 final release date.
It fabricated sources. Q3 cited a nonexistent Impress press release URL (404), and gave a different wrong date each time.
It barely searched at all. Average 1.7 times, less than half of K3's 4.0.

Different model sizes — this isn't a blanket verdict on Qwen. But it demonstrates one thing: web search accuracy isn't determined by the search API alone.

Local Qwen's insistence that "it's currently 2024" is typically classified in model evaluation as temporal hallucination, and its causes differ from ordinary factual hallucination. A large language model's knowledge cutoff date determines the latest point in its training data, but the model has no built-in clock telling it what today's date is. Unless the current date is explicitly injected into the system prompt, the model can only infer time from two clues: the temporal distribution of its training data, and the current conversation context. When the context has no explicit date, the model falls back to the "most recent feeling" in its training data, and 2024 becomes its default anchor. The danger is that this internal anchor overrides external evidence — even if search results clearly show a 2026 date, the model may judge "that's a future date, impossible" and refuse to accept it. This differs from "not knowing the answer" — the model is actively rejecting correct information. Mitigation measures include: hard-coding the current date in the system prompt, requiring the model to restate retrieved timestamps before answering, and avoiding letting the model judge "what time it is now" on its own.

Finding 3: Kimi Search API's Two Gaps

All three of K3's errors were on q4. It tried 12 different phrasings and never found the article published that day. The cause wasn't the model — Kimi's search index simply hadn't indexed that page yet.

The second gap is more serious: queries containing the site: operator return HTTP 502. Both K3 and Qwen tried site:kenimoto.dev to locate a specific page, and every time the search API returned a non-JSON error page — the first time it even crashed the script.

Calling the search API directly also exposed the importance of phrasing. Searching "Kimi K3 release date Moonshot AI," the word "date" hijacked the results, returning timezone websites and JavaScript Date documentation. Searching just "Kimi K3" put the official site, paper, and GitHub at the top. In the actual runs, though, none of the models fell into this trap — both K3 and Qwen adjusted their queries on their own.

Incidentally, the soon-to-be-retired $web_search tool is incompatible with K3 — the second call (the one returning search results) reported 400 tokenization failed every time. Passing search through ordinary function calling, as this test did, works fine.

Claude's Japanese Book Struggle

Q3 asked for the publication date of a Japanese book, and Claude said in 2 of 3 runs that it couldn't find the book and it might not exist. In fact, the book is listed on the publisher's NextPublishing page and on Rakuten Books, complete with publication date.

Claude did show resilience elsewhere. In one q4 run, when WebSearch failed to retrieve the article, Claude switched to finding it through the sitemap and opened the page to confirm the title and date. Since WebSearch only returns titles and URLs, on-demand page fetching is part of the design, and here it paid off. Claude also frequently noted that it had only consulted secondary sources to avoid drawing conclusions. The 22-second median latency includes Claude Code's agent loop; calling the search tool directly via API should be faster.

Practical Recommendations

For research and fact-finding, keep cached search as your default. If you use Codex, you don't need to add --search every time. The safer setting doesn't sacrifice accuracy.
For non-English sources, don't trust "not found." Not finding something doesn't mean it doesn't exist. When Claude says it can't find it, try another approach.
If you're integrating search yourself, the model matters more than the search API. With the same search API, K3 and Qwen differed by more than double. Pick a model that can read search results first, then pick the cheapest search API.
Make your search tool return failure information. Kimi's search API returns 502 on site:. Pass failures back to the model instead of letting the script crash — the model will rephrase and search again.

Limitations

This is a small-scale test: 8 questions, 3 runs. The topics skew toward Kimi K3 and MCP, and q3 and q4 are my own book and article. Claude ran through Claude Code, and Codex results came from calling the CLI's underlying API. The Qwen configuration was a local 35B model connected to Kimi search, not Qwen's official search. Each configuration works differently, latencies aren't directly comparable, and total cost wasn't calculated.

Next, I'd like to call Claude's search tool directly via API and try Qwen's official search on Alibaba Cloud, to narrow the gaps between configurations.

The question of search has shifted from "which engine" to "where does search run, and who invokes it." That change happened faster than anyone expected four months ago.

One-line Summary

120 tests show search accuracy depends not on the underlying engine, but on where search runs and who interprets the results.

Paragraph Summary

The article compares four search configurations — Claude, Codex, Kimi K3, and local Qwen — across 120 evaluations, arguing the industry focus has shifted from "which search engine" to "where search runs and who invokes it." Tests covered eight categories including news, date traps, Japanese sources, and documentation details, each run three times. Results were surprising: OpenAI's cached search got all 24 questions right, showing caching doesn't sacrifice freshness — a same-day article was indexed within three hours. More critically, K3 and local Qwen used the identical Kimi search API yet scored 21 to 9, proving search accuracy is determined mainly by the model's ability to interpret results, not the search API itself. The article also reveals Kimi's search index freshness gap, site: operator returning 502, and Claude's weaker Japanese-source retrieval, offering practical advice like making cached search the default and not trusting "not found."