Burning 300M Tokens to Test DeepSeek Search API: A Five-Round Showdown with Claude Search

Title

Burning 300M Tokens to Test DeepSeek Search API: A Five-Round Showdown with Claude Search

Meta Description

A Bilibili UP主 self-funded 300M tokens to test DeepSeek^Ad Search API vs. Claude Search in a five-round showdown: from accuracy and small-language performance to speed, cost, and honesty, revealing real data where DeepSeek Search wins three rounds and Claude models win all. All tests are open-source and reproducible.

Article Content

Whether an AI answer is accurate depends half on the model itself and half on the search capability it invokes. The model is the brain that answers questions, while search is the assistant that flips through materials for it. When DeepSeek started selling its own search API separately at a price of only 5 yuan per 1000 calls, a key question surfaced: is it really better than Claude's built-in search?

Bilibili UP主 "懒得理你" (Too Lazy to Care) self-funded 300 million tokens to conduct the first open-source evaluation comparing DeepSeek Search and Claude Search together. The test covered not only Chinese and English but also small languages such as Japanese, Korean, and Thai, unfolding five rounds of showdown across five dimensions: accuracy, speed, cost, and honesty. This article combines measured data and screenshots to dissect the real results of this search API battle.

Test Design: Fair Comparison and Open-Book Rules

For a fair comparison, the evaluation set up a rule similar to an exam: each model first answers closed-book (no search allowed), then answers open-book (search allowed). In open-book mode, up to five searches are allowed, and the model can only view the returned results and summaries, not click into the original web pages, purely to compare the quality of information delivered by the search itself. Scoring rules: correct answer +5 points, wrong hard submission -3 points, no answer no penalty, and from the second search onward, -1 point per search.

The protagonist chosen was Claude, because it was the strongest tier model at the time and could be configured with both search types—Claude's search is not offered externally, while DeepSeek's search can be used with any model. The test pitted Claude Sonnet and Opus against DeepSeek's Flash and Pro tiers, with ten sets of questions and one hundred questions in total, half Chinese and half English, mostly from public question banks from Google and Alibaba Tmall, with the rest self-composed cold knowledge and recent news. Grading was blind, with one judge from Claude and one from DeepSeek, plus one from OpenAI's GPT, best two out of three votes, and even the exam program was written by GPT.

Open-book/Closed-book Testing: Drawing on the open-book and closed-book exam formats in educational assessment, closed-book means the model answers using only internal knowledge, while open-book allows the model to call search tools to obtain external information. This design separates the model's inherent knowledge from its ability to utilize external information. Blind Grading: Means judges score without knowing which model generated the answer, to avoid bias, similar to double-blind experiments in academic review. Best Two Out of Three Votes: Uses a majority voting mechanism, with three independent judges (Claude, DeepSeek, GPT) each evaluating, and the majority result taken as the final judgment, improving scoring objectivity.

Round 1: Accuracy—Huge Gap After Open-Book

After excluding eight ambiguous questions, calculations were based on 92 questions. In closed-book, the four models answered 60 to 73 correctly, and news questions from the last month all scored zero. Once open-book, all models answered 87 or more correctly. The cheapest Flash, after open-book, answered 18 more correctly than the most expensive Opus in closed-book.

The same Sonnet, using its own search, got five wrong, but after switching to DeepSeek Search, got all correct. One question asked "Who is the director of Letters to Komomo", and Claude's search returned content about love letters and failed; DeepSeek Search found director Hiroyuki Okiura in one search. However, the difference was only these five questions, so luck couldn't be ruled out. The higher-end Opus was more straightforward, getting all correct with both searches. But Claude's search averaged more searches, and after scoring, DeepSeek narrowly won.

Why did Claude search more? Because Claude's search first hands results to its own small model to read, returning only a summary of about 2000 characters; DeepSeek directly returns ten results, over 10,000 characters. Less material naturally requires more searches. But the first round's questions were too easy, with open-book almost all correct, so the score gap didn't widen.

Differences in Search API Return Mechanisms: Claude's search first has an internal small model summarize search results, returning only about 2000 characters of condensed information; while DeepSeek Search directly returns ten raw results, totaling over 10,000 characters. This difference causes Claude Search to require more calls to gather enough information, but summaries may lose details; DeepSeek Search provides a large amount of information in a single call but may include redundant content. Token Consumption: In AI models, tokens are the basic units of text processing, and the number of characters returned by search directly affects token consumption, thereby affecting processing cost and speed.

Round 2: Small Languages—DeepSeek Search Clearly Advantageous

The second round switched to a harder multilingual question bank, from a multilingual dataset publicly released in July of that year, with questions based on local materials rather than translated from English, covering Japanese, Korean, Thai, Indonesian, Russian, and Spanish, twenty each. During trial runs, the DeepSeek account once reported an error due to insufficient balance, showing the consumption was large, so GPT was invited as judge for the formal exam. After excluding twenty-five ambiguous questions, calculations were based on 95 questions.

The difficulty surged: in open-book, the four models only answered 36 to 56 correctly. With Claude open-book, Sonnet using its own search answered 73 correctly, but after switching to DeepSeek Search, soared to 85; Opus went from 83 to 90. The biggest gap was Japanese: the four examinees using DeepSeek Search got all twenty correct, while those using Claude Search got only fifteen or sixteen. Sonnet's score nearly doubled.

Round 3: Speed and Cost—Each Wins Some

The same DeepSeek Search given to both companies' models is another competition. High-end vs. high-end, small language scenarios, Claude's Opus answered 90 correctly, DeepSeek's own Pro answered 83; mid-tier Flash answered three more correctly. But Flash averaged two and a half searches, Sonnet only one and a half, and after scoring, Sonnet won. All four scoring scenarios were won by Claude's models, and the one where DeepSeek Search won had only one judge, so the UP主 later rechecked with multiple agents for two more rounds.

In speed, the same Sonnet using Claude's own search took twenty to thirty seconds per question, but with DeepSeek Search only a little over ten seconds, because Claude Search has the extra step of a small model writing summaries. This round DeepSeek won.

In cost, at list price, one search with Claude Search costs thirteen times that of DeepSeek. But after Sonnet switched to DeepSeek Search, each question became over twenty percent more expensive—because developers pay Claude by the number of characters read, and DeepSeek delivers several times more material, so the money saved on search is all spent on reading material. With Opus, it's over fifty percent more expensive. A cheaper combination is DeepSeek model with DeepSeek Search: Sonnet costs over fifty cents per question, Flash only four cents. This round is a tie—DeepSeek Search has a cheap unit price but expensive processing, Claude Search has an expensive unit price but cheap processing.

Round 4: Honesty—Claude Has Fewer Hallucinations

The final round compares honesty: when a question is unknown, does it directly say it doesn't know, or does it make something up? The director question for "Letters to Komomo" at the beginning: Claude Search didn't find it and directly said it couldn't find it, without substituting another film. In closed-book, the evaluators instructed that if you don't know, say you don't know, but wrong answers come in several types: directly saying you don't know, answering half and guessing one and saying it might not be right, and confidently answering wrong—the last is what's commonly called hallucination.

Combining both rounds, the number of confidently wrong answers: Opus one, Sonnet four, Flash eighteen, Pro twenty-six. This round Claude won.

Hallucination: Refers to AI models generating plausible but actually incorrect or fabricated information, a key indicator for evaluating model reliability. In search-augmented scenarios, models may hallucinate due to misleading search results or their own knowledge deficiencies. Honesty Evaluation: Quantifies model honesty by distinguishing response types when uncertain (directly admitting not knowing, guessing but stating uncertainty, confidently answering wrong), with confidently wrong answers being typical hallucinations and most harmful to user experience.

Five-Round Summary and Unmeasurable Boundaries

After five rounds: in the four rounds comparing search, DeepSeek won three rounds by a few cents and tied one; in the model附加赛 and honesty, Claude won all. The highest scores in two rounds were achieved by Claude's Opus paired with DeepSeek Search.

The UP主 also clarified what couldn't be tested: some small-language questions actually originated from Chinese or English web pages, which doesn't prove DeepSeek's small-language web coverage is more complete; both companies' models and searches are updating, and this is only one result from October, with each question tested only once; not allowing clicking into web pages actually disadvantages Claude Search, which normally clicks into pages to read originals; in terms of price, Flash is closer to Claude's Haiku, but Haiku already helps write summaries in Claude Search, so it wasn't tested separately this time.

All questions, exam programs, and raw records are open-sourced, with links in the video's pinned comment. The value of this evaluation isn't in settling the debate between the two searches, but in providing a reproducible evaluation framework—at a time when AI search APIs are gradually becoming infrastructure, developers need not just unit price, but a comprehensive trade-off among search quality, processing cost, and honesty.

One-line Summary

Bilibili UP主 self-funded 300M tokens to test DeepSeek vs. Claude search APIs, five-round showdown reveals each's pros and cons.

Paragraph Summary

This article introduces an open-source evaluation initiated by a Bilibili UP主, comparing the performance of DeepSeek Search API and Claude's built-in search. The test used a closed-book/open-book five-round system, covering multiple languages including Chinese, English, Japanese, Korean, and Thai, unfolding across dimensions like accuracy, speed, cost, and honesty. Results show: DeepSeek Search has clear advantages in small-language scenarios, with unit price only one-thirteenth of Claude's, but large returned material leads to higher processing costs; Claude Search is better in honesty with fewer hallucinations. Ultimately, DeepSeek Search won three rounds and tied one in the four search comparisons, while Claude models won all in附加赛 and honesty. The evaluation provides a reproducible framework, emphasizing the importance of comprehensively weighing search quality, cost, and honesty.