SerpApi Achieves 97.6% Accuracy on SimpleQA Verified

Article Content

Amid growing attention to AI agents' search capabilities, search engine API provider SerpApi has published an internal evaluation result: after integrating its Google Search API and Extract API with the GPT-6.1 Sol model, it achieved 97.6% accuracy on the SimpleQA Verified benchmark introduced by Google DeepMind and Google Research, tying for first place with Tavily's published score. This result not only demonstrates the potential of search tools working in tandem with models but also provides a new reference point for AI agents' factual question-answering capabilities.

Evaluation Background and Challenges

SimpleQA Verified is a benchmark specifically designed to test the accuracy of AI models in answering factual questions. It contains 1000 questions selected from OpenAI's original SimpleQA dataset after reference-answer verification, removal of similar questions, and topic balancing. The difficulty of this benchmark lies in the fact that many questions require the model to retrieve and integrate information from multiple sources rather than relying solely on parametric knowledge.

SerpApi's testing goal was precisely to explore how Google search results and web content can help AI agents search the web and answer factual questions. They built an agent system that allows the model to autonomously decide when to search and when to read pages, and simulated real-world scenarios through search and tool-call limits.

SimpleQA Verified is derived from the SimpleQA dataset released by OpenAI in 2024, which contains approximately 4000 human-written factual questions with unique and short answers, designed to expose large models' tendency to hallucinate on long-tail knowledge. Because the original version had annotation errors, duplicate questions, and imbalanced topic distribution, Google DeepMind and Google Research performed reference-answer verification, similar-question deduplication, and topic balancing, ultimately retaining 1000 high-quality questions to form SimpleQA Verified. Scoring for this benchmark typically uses strict exact match, requiring the model to give a short answer consistent with the reference answer rather than a vague or partially correct statement, thus imposing high precision requirements on retrieval-augmented agents.

Testing Methodology: Synergy of Search and Extraction

In the test setup, the answer model used GPT-6.1 Sol (low reasoning effort), the search step used SerpApi's Google Search API (engine=google), and search results were output in Markdown format. When the model needed more details, it could call the Extract API in scrape mode to obtain the Markdown content of web pages. Each question allowed multiple searches and scrapes, but each scrape was limited to 48000 characters, and each model input was limited to 56000 tokens.

This design lets the agent behave like a human researcher: first locate relevant pages through Google SERP data (including organic results, answer boxes, and AI Overviews), then read page content in depth, and perform supplementary searches when necessary. SerpApi specifically removed metadata and pagination details and used no_cache=true to ensure fresh results on each run.

Scoring was performed by GPT-5.6 Sol (high reasoning effort), comparing the reference answer and predicted answer for each question without access to search tools. You may not have noticed that SerpApi also conducted benchmark leakage detection, reviewing the search results and page content seen by the model to exclude 40 questions that exposed benchmark answers or evaluation prompts, ultimately answering 937 of the remaining 960 questions correctly.

The scrape mode of the Extract API is a web content extraction capability provided by SerpApi. Unlike returning only search snippets, scrape mode actually fetches the full HTML of the target page and converts it into structured Markdown text, preserving semantic structures such as headings, paragraphs, lists, and tables while removing noise such as scripts, styles, and navigation. This allows the model to read the page body directly without parsing raw HTML itself. The 48000-character scrape limit and the 56000-token model input limit together constrain the depth of a single reading, prompting the agent to make trade-offs within limited context and initiate new searches or scrapes when necessary, thereby simulating the stepwise retrieval behavior of human researchers in an information-overloaded environment.

Benchmark leakage is an easily overlooked systemic risk in retrieval-augmented evaluation. When a model performs searches or scrapes, the returned web content may directly contain the benchmark's reference answers, original questions, or evaluation prompts, allowing the model to answer without genuine reasoning and thus inflating accuracy. SerpApi's leakage detection identifies and excludes these contaminated samples through manual or rule-based review of the search results and page content actually seen by the model. A total of 40 questions were removed this time, and results were ultimately reported on 960 questions. Although this practice reduces the evaluation scale, it significantly improves the credibility of the scores and provides a reproducible evaluation standard for other search API vendors and agent developers.

Results Comparison: Tied for First with Tavily

SerpApi compared its own results with the published evaluation reports of Tavily and Parallel. The data shows: SerpApi Google + Extract tied for first with 97.6% accuracy alongside Tavily Search, followed closely by Parallel Advanced at 97.0%, Perplexity Search at 95.0%, and Exa Auto at 91.0%.

As a side note, these scores come from independent evaluations under different settings, so they are more for contextual reference than direct comparison under strictly identical conditions. SerpApi also explicitly stated that it only tested its own product, and the scores of other vendors come from public reports by Tavily and Parallel.

Next Steps: SealQA and More Complex Challenges

SerpApi is not stopping here. It is evaluating SealQA, a benchmark that includes questions with conflicting or useless search evidence, placing higher demands on agents' information discrimination capabilities. In the completed Seal-0 run, using GPT-6.1 Sol (medium reasoning effort) with SerpApi Google and Extract, the score was 58.56% (65/111). Full SealQA results and methodology will be shared in a follow-up article.

The significance of this evaluation result lies in validating that combining search APIs with models can significantly improve the accuracy of factual question answering. For developers building AI agents, choosing appropriate search tools and web extraction solutions may be more effective at solving hallucination problems than simply increasing model scale. SerpApi's practice shows that an agentic workflow that teaches the model "when to search and when to read" is an important path toward reliable AI applications.

As benchmarks like SimpleQA Verified become more widespread, competition among search API vendors will shift from mere result coverage to deep support for agent workflows. The details disclosed by SerpApi this time, including character limits, token limits, and leakage exclusion methods, also provide the industry with a reproducible evaluation paradigm.

SealQA is a benchmark aimed at highly difficult information retrieval and reasoning, whose core design lies in the fact that the search evidence involved in its questions is often conflicting, outdated, or irrelevant to the question. Models cannot simply rely on top-ranked results; they need to judge source credibility, identify contradictory information, and decide when to abandon unreliable evidence. Seal-0 is a subset of SealQA containing 111 questions, and SerpApi achieved a score of 58.56% with medium reasoning effort, far below the 97.6% on SimpleQA Verified. This gap shows that when the search environment shifts from "information sufficiency" to "information noise and conflict," agents' information discrimination and evidence weighting capabilities still face significant challenges.

One-line Summary

SerpApi combined GPT-6.1 Sol with search/extract APIs to achieve 97.6% accuracy on SimpleQA Verified, tying for first with Tavily.

Paragraph Summary

SerpApi published internal evaluation results: after integrating the Google Search API and Extract API with the GPT-6.1 Sol model, it achieved 97.6% accuracy on the SimpleQA Verified benchmark, tying for first with Tavily. The test built an agent system allowing the model to autonomously decide when to search and when to read pages, and simulated real-world scenarios through search and tool-call limits. Scoring was done by GPT-5.6 Sol without search tools, and 40 benchmark leakage questions were excluded, ultimately answering 937 of 960 questions correctly. Comparisons show Parallel Advanced at 97.0%, Perplexity Search at 95.0%, and Exa Auto at 91.0%. SerpApi's next step is to evaluate SealQA, which includes conflicting evidence; the completed Seal-0 score was 58.56%.