Building AI Agents in Rust: The GAIA Benchmark Reveals the Capability Ceiling of Large Models

In the process of building AI Agents, many developers get used to spending huge amounts of time repeatedly fine-tuning prompts, yet they often can't tell whether these efforts actually lead to real progress. It's like sailing without a compass—when you don't know your direction, the more you invest, the more likely you are to drift off course. Based on Episode 4 of the Rust AI Agent tutorial series, this article uses one of the industry's widely recognized high-difficulty benchmarks—GAIA—to explore why tools are indispensable for AI Agents.

The GAIA Benchmark: Why Tools Are Indispensable

GAIA (General AI Assistants) is an authoritative benchmark specifically designed to evaluate general-purpose AI assistants. Jointly developed by Meta AI Research, HuggingFace, and AutoGPT, it was released in 2023. Unlike the math reasoning or code generation benchmarks that models such as GPT-4 excel at, GAIA specifically tests an AI's comprehensive ability to complete assistant-type tasks in the real world. Its design philosophy is: a truly general-purpose AI assistant needs not only a knowledge base, but also the ability to operate tools, perform multi-step planning, and remain robust in uncertain environments.

GAIA's emergence had a specific historical backdrop. Between 2022 and 2023, as large models like GPT-4 and Claude approached or even surpassed human-level scores on traditional academic benchmarks such as MMLU and HumanEval, the research community began to realize that these benchmarks could no longer effectively distinguish a model's true assistant capabilities. Notably, MMLU (Massive Multitask Language Understanding) covers over 57,000 multiple-choice questions across 57 subject domains, while HumanEval focuses on function-level correctness verification for code generation—both are "static" assessments within a closed knowledge space, unable to measure a model's ability to dynamically interact with the external world. GAIA's core innovation lies in treating "tool use" and "multi-step planning" as first-class citizens in its evaluation design, rather than viewing them as optional bonus points. This orientation profoundly influenced the design thinking of subsequent Agent evaluation frameworks—benchmarks like WebArena and AgentBench all adopted a similar "task-completion-oriented" evaluation paradigm. WebArena focuses on web operation tasks, requiring Agents to complete tasks such as shopping and information retrieval in a real browser environment; AgentBench, on the other hand, builds a unified evaluation platform spanning 8 different environments including operating systems, databases, and web browsing. Together, these benchmarks constitute the mainstream ecosystem of current Agent capability evaluation, and GAIA, as the benchmark most closely aligned with general assistant scenarios, holds the broadest reference value in industry.

In its scoring mechanism, GAIA adopts a strict Exact Match principle: an answer is either completely correct or scores zero, with no partial credit. This design deliberately avoids scoring ambiguity, but it also means that even if a model's reasoning process is entirely correct and it errs only in a single digit, it still cannot earn any points. This is starkly different from the fuzzy-matching practices allowed by many academic benchmarks, and it aligns more closely with the real-world business need where "a result is either right or wrong." Another notable characteristic of GAIA questions is that they are relatively simple for humans (with an average human accuracy of about 92%), yet extremely challenging for top-tier AI models, fully exposing the weaknesses of current large models in tool use and long-horizon reasoning. Their defining feature is that tasks closely resemble real-world scenarios. To illustrate the necessity of tools, let's first look at a classic GAIA Level 1 example:

If marathon superstar Eliud Kipchoge could run indefinitely at the pace at which he broke the world record, how many thousand hours would it take him to run the closest distance between the Earth and the Moon at perigee?

This question looks simple, but for a pure large language model, it's precisely where the pain lies. The model needs to: first, know Kipchoge's world-record pace; second, know the precise value of the Moon's perigee distance (from Wikipedia); and finally, perform high-precision division of large numbers.

Without a search engine and calculator, the model can only rely on its training corpus to "guess" these numbers. Even if it remembers the pace, once it misremembers the Wikipedia value or miscalculates a single digit in the large-number division, then according to GAIA's scoring rules, this question scores zero outright. This fully demonstrates that even the simplest Level 1 task requires the coordinated use of at least three types of tools: web search, file lookup, and calculator.

It's worth noting that while prompt engineering has accumulated many techniques such as Chain-of-Thought, Few-shot Prompting, and Self-Consistency, these methods are essentially still optimizations within the model's existing knowledge and capability boundaries. Chain-of-Thought reduces skip-step errors by having the model unfold its reasoning process step by step; Few-shot Prompting guides output format by providing examples in the context; Self-Consistency improves stability by sampling multiple times and taking the majority answer. However, these three types of techniques share one fundamental limitation: they cannot enable the model to access information after its training cutoff date, cannot guarantee the precision of floating-point arithmetic, and cannot handle external files the model has never seen.

This limitation is more intuitively understood from an information-theoretic perspective: after a large language model completes training, its "knowledge" is compressed and encoded into the weight matrices of billions of parameters. This lossy compression process inevitably discards details such as precise values and the latest events, while retaining statistical regularities such as language structure and conceptual relationships. Specifically, a 7-billion-parameter language model's weight file occupies about 14GB of storage space, while its training corpus often reaches several TB or even tens of TB—meaning each parameter must, on average, "compress" thousands of bytes of raw text information. Under such an extremely high compression ratio, what the model excels at distilling is order-of-magnitude perception like "the marathon world record is roughly around 2 hours," rather than precise values like "2:01:09"; what it excels at expressing is conceptual relationships like "the distance between the Moon and the Earth varies due to orbital changes," rather than precise quantities like "the perigee is about 356,500 kilometers." This means the model is naturally suited for semantic understanding and pattern recognition, rather than precise numerical storage and real-time information retrieval—the latter being exactly where tools come into play. When a task requires accessing information after the model's training cutoff, precise numerical computation, or handling external files, no matter how sophisticated the prompt is, it is of no avail. This is precisely the fundamental reason architectures such as Tool Use and Retrieval-Augmented Generation (RAG) emerged—they expand the model's capability boundaries at the architectural level rather than the prompt level. RAG supplements the model's static knowledge by dynamically retrieving external knowledge bases at inference time; Tool Use grants the model interfaces to call external capabilities such as calculators, search engines, and code interpreters. Together, they form the infrastructure layer of modern Agent architectures.

GAIA's Three-Tier Difficulty Classification

GAIA divides tasks into three difficulty levels:

• Level 1 (Basic): Basic fact lookup and simple reasoning, usually requiring only 0-1 tools, with fewer than 5 thinking steps. This is the baseline level for this test.
• Level 2 (Intermediate): Requires combining multiple different tools, with 5-10 steps.
• Level 3 (Expert): No step limit; may invoke arbitrarily many tools, requiring the Agent to perform long-horizon planning across complex web pages and local files like a real human assistant.

The design logic of this three-tier classification reflects two core dimensions of Agent task complexity: the breadth of tool invocation (how many different tools are needed) and the depth of the reasoning chain (how many steps are required to arrive at the answer). The performance gap between Level 1 and Level 3 often directly reflects an Agent system's engineering maturity in Tool Orchestration and Long-term Memory—the former determines whether the Agent can flexibly switch tool types within a single task, while the latter determines whether the Agent can maintain contextual coherence across dozens of reasoning steps without suffering "forgetting drift." Current top-tier commercial models typically score no more than 10% on Level 3—a stark gap compared to the human level of 92%. This chasm is precisely the core challenge for the Agent engineering field in the coming years.

For developers, the significance of starting benchmark testing at Level 1 is to see clearly how many tasks a pure large model can actually complete under conditions where "all tools are stripped away."

Building an AI Agent Evaluation Framework in Rust

The overall approach for this evaluation is clear: load the GAIA Level 1 validation set from Hugging Face (taking the first several questions as a small-scale sample), concurrently call multiple models such as GPT, Claude, and DeepSeek^Ad via API, parse the data into structured form, and finally perform exact-match verification against the standard answers.

Rust's advantages in AI Agent engineering are mainly reflected on two levels. First, the async runtime centered on Tokio is naturally suited for handling scenarios with large numbers of concurrent API calls, avoiding the GIL (Global Interpreter Lock) limitations of Python's asyncio in complex scenarios. The GIL is a mutex in the CPython interpreter that ensures only one thread executes Python bytecode at any given moment. This makes Python multithreading nearly unable to leverage multi-core advantages in CPU-intensive tasks, whereas Rust's async/await combined with Tokio's multi-threaded scheduler has no such limitation.

Understanding how Tokio works helps you better leverage Rust's concurrency advantages. Tokio uses an M:N threading model: a small number of OS threads (usually equal to the number of CPU cores) run a large number of lightweight "Tasks." Each async task voluntarily yields execution when waiting for I/O, and the scheduler then allocates the CPU to other ready tasks, thereby achieving high concurrency on a single thread. This shares the same design philosophy as Go's goroutines and Erlang's processes, but Rust guarantees memory safety through compile-time ownership checking with zero runtime overhead, without the intervention of a garbage collector (GC). For Agent evaluation scenarios that need to make requests to multiple LLM APIs simultaneously, this means hundreds of concurrent HTTP requests can run efficiently with extremely low memory footprint. More concretely, Tokio's primitives such as JoinSet and FuturesUnordered allow the pattern of "concurrently launch N API requests, wait for all results, and aggregate statistics" to be elegantly implemented in just over a dozen lines of code, whereas equivalent Python code often requires introducing multiple dependencies like asyncio.gather and aiohttp and handling various edge cases. This engineering simplicity is especially valuable in research scenarios that require frequently adjusting concurrency parameters and rapidly iterating on evaluation schemes.

Second, the strong type system and ownership model ensure memory safety when serializing/deserializing JSON Schema and handling API responses, reducing runtime errors caused by type mismatches. For Agent services that need to run stably over the long term, Rust's zero-cost abstractions and extremely low runtime overhead also make it a strong contender for production-grade Agent infrastructure.

Data Preparation and Type Definitions

First, you need to obtain an access Token on Hugging Face. After creating it, store it in the HF_TOKEN environment variable in the .env file. Then define the core data types corresponding to the GAIA API in the Rust project:

• HuggingFaceResponse: contains a rows vector collection
• GaiaRow: contains key fields such as task_id, question, level, and final_answer
• GaiaOutput: the output format returned by the large model according to a fixed JSON Schema
• GaiaEvaluationResult: the evaluation result, containing fields such as the question ID, the model used, whether it's correct, the standard answer, and whether an error occurred

Regarding the JSON Schema design of GaiaOutput, it's worth a dedicated explanation: constraining output with JSON Schema (Structured Output) is a key technique for making the LLM return a machine-parseable format. In 2024, OpenAI officially introduced the response_format parameter, allowing developers to pass in a JSON Schema to force the model to return data in the specified structure, fundamentally solving the format instability problem of earlier approaches that requested JSON output via prompts. At the underlying implementation level, OpenAI's Structured Output is based on Constrained Sampling: at each decoding step of the model, only tokens conforming to the current JSON Schema constraints are allowed to be sampled, thereby mathematically guaranteeing the validity of the output format, rather than relying on the model's "conscientious" compliance. The technical principle of constrained sampling can be further understood as follows: at each decoding step, the language model computes a probability distribution over the entire vocabulary (usually containing tens of thousands of tokens), and normally samples the next token according to probability. In constrained sampling mode, the system pre-builds a Finite State Machine based on the JSON Schema, and at each decoding step forcibly sets the probability of any token that violates the current state's constraints to zero, so that sampling can only proceed from valid tokens. This approach guarantees format correctness while having almost no negative impact on the model's semantic generation ability, because in most cases, the semantically reasonable next token happens to also be the format-valid token. The is_solved field is used to determine whether the model claims to have provided an answer, the final_answer field is used for exact-match scoring, and the reasoning field can be used for subsequent error analysis. Claude's tool_use mechanism and DeepSeek's response_format also provide similar capabilities, but the specific implementation details differ and require an adaptation layer for unified handling.

Loading the Dataset

Loading the Level 1 dataset requires sending an HTTP request to Hugging Face. Here we use the reqwest library with its json feature enabled. reqwest is the most mainstream async HTTP client in the Rust ecosystem, and combined with serde_json it can elegantly handle structured data. The core function reads the Token via an environment variable, uses reqwest's client to send a GET request, specifies the dataset as GAIA with the configuration 2023_level1, split as the validation set, and carries Bearer authentication.

Problem-Solving Logic and Fault-Tolerance Design

The problem-solving function solve_problem takes three parameters—the model, the system prompt, and the user prompt—prepares the JSON Schema output format, and sends the request via the async OpenAI client. There is a key design here—checking whether the model refuses to answer. Due to safety policies, the model doesn't necessarily answer all questions. If the model stops generating for safety reasons, a special GaiaOutput is generated, setting is_solved to false and marking it as "model refused to answer."

To improve stability, the author wraps solve_problem in a retry layer. Exponential Backoff is a standard strategy for handling transient failures in distributed systems: the waiting time before each retry grows exponentially (e.g., 1s, 2s, 4s), and Jitter is usually added to avoid a large number of clients retrying simultaneously and causing server congestion (i.e., the "thundering herd" effect). In AI Agent scenarios, this strategy is especially important because LLM API services usually have strict rate limits and occasional 503 timeouts. Understanding the two dimensions of rate limiting helps in designing a more reasonable retry strategy: RPM (Requests Per Minute) limits the request frequency, while TPM (Tokens Per Minute) limits data throughput. When evaluating multiple GAIA questions, even if a single request doesn't trigger the RPM limit, if each question's reasoning chain is long, the accumulated TPM consumption may still trigger throttling. Therefore, production-grade Agent systems usually need to monitor both dimensions simultaneously and, in the retry logic, distinguish between the two different types of errors: "429 Too Many Requests" (which requires waiting) and "503 Service Unavailable" (which can be retried immediately). The combination of Rust's tokio::time::sleep and async/await makes implementing this pattern both concise and non-blocking to threads, with up to three retries. This precisely embodies Rust's engineering mindset for building robust systems.

Evaluation Logic and Empirical Results

The evaluation-related code contains two core functions. The function that determines correctness has concise logic: if the answer result is empty (possibly a refusal to answer), it's counted as wrong; otherwise, the result is trimmed, converted to lowercase, and exact-matched against the standard answer.

The complete evaluation workflow is built into a standalone binary (gaia.rs): download the questions, loop through each model, execute concurrently for each question, collect results, and finally print the accuracy.

A noteworthy engineering detail is that free models have very low concurrency limits. It's recommended to directly use DeepSeek or lower-tier commercial models on OpenRouter for testing, to keep costs under control. At the same time, it's essential to load the .env environment variables and configure tracing in the main function, otherwise you'll run into issues where the environment variables can't be found or results aren't printed.

Empirical Data Comparison Across Major Models

In a small-scale local test of 10 questions, Claude Haiku 4.5 achieved 30% accuracy and GPT-4.1 mini achieved 20%. Below are the results from a larger-scale test of 20 questions:

• GPT-5: 50% (suspected training set leakage, data unreliable)
• GPT-5 mini: 20%
• Claude 3.5 Sonnet / Haiku: about 15%
• Other mainstream models: around 20%, at most about 25%

GPT-5's result of as high as 50% is not reliable. Training Data Contamination is a core controversy in the current AI benchmark evaluation field: when a public benchmark dataset is incorporated into a model's training corpus, the model may answer questions through "memorization" rather than "reasoning," leading to inflated evaluation results that lose reference value. Contamination detection itself is also an active research field, with common methods including: comparing the performance difference of a model on public sets versus private holdout sets, detecting whether a model can "complete" the latter half of a test question's text, and analyzing the model's performance decay curve on benchmarks released at different times.

The root of this problem lies in the scale and breadth of modern large model training corpora—datasets like Common Crawl gathered from web crawlers encompass almost all publicly released text on the internet, including academic papers, GitHub repositories, and the questions and answers of various evaluation datasets. Model developers usually attempt Decontamination, but due to the enormous scale of training data, completely eliminating contamination is extremely challenging from an engineering standpoint. Academia has proposed various countermeasures, the most promising of which is the design philosophy of Dynamic Benchmarks: platforms like LiveBench automatically generate new questions each month from the latest published papers, news events, and competition problems, ensuring that the evaluation data always postdates the model's training cutoff; in addition, the "Canary Test" method embeds special marker strings into the test set—if a model can reproduce these strings, contamination can be confirmed; there's also Adversarial Data Construction, which distinguishes "comprehension-based answering" from "memorization-based answering" by applying semantically equivalent transformations to the original questions (such as changing numbers or substituting proper nouns). This issue reminds developers to remain cautious when interpreting model leaderboards—as models iterate, the evaluation validity of already-published benchmark datasets decays over time. After excluding the outlier GPT-5, the actual capability ceiling of other models is about 25%, a figure that carries more reference value.

Three Iron Rules: The Capability Boundaries of Large Models

This benchmark test yields three important insights:

First, large models know when they need tools. From the behavior of models choosing to refuse to answer rather than "pretending to know" when they can't solve a problem, we can see that models possess a degree of self-awareness. Developers can leverage this by having the model proactively initiate tool-invocation requests within the Agent architecture when it discovers missing information.

Second, no matter how large the training data scale is, it cannot replace tool invocation. Real-world applications require 100% accuracy and up-to-date data, and this can only be achieved through actual code that reads files and accesses web pages—model memory alone cannot meet these needs.

Third, in real-world scenarios, about 75% of tasks depend on external tools. Even for what appears to be the simplest GAIA Level 1 tasks, up to 75% of cases require external tools to complete. Analyzing the incorrect answers reveals that a considerable portion of questions require web search and web content retrieval, while another portion requires reading and parsing external files. This 75% figure aligns closely with the practical experience of industry Agent systems in recent years—enterprises such as Salesforce and ServiceNow have generally found, when deploying production-grade Agents, that the proportion of tasks that can truly be handled by a "pure LLM" is extremely low, and the vast majority of business scenarios require Agents to have tool capabilities such as accessing CRM systems, calling internal APIs, and reading document attachments. This also explains why, over the past two years, mainstream large model vendors have made "tool invocation capability" one of the core selling points of their model releases, and have provided increasingly refined Function Calling interface specifications in their corresponding APIs.

Conclusion: Breaking Through the 25% Capability Ceiling

This benchmark test clearly delineates the capability boundary of pure large models: an accuracy ceiling of about 25%. And the advantage of Rust developers lies precisely in their engineering capabilities in async network handling and type safety.

The next step is to leverage these advantages of Rust to build robust web search components and lightweight computation tools for AI Agents, thereby breaking through this 25% ceiling at the architectural level. This also confirms the core consensus in current Agent development: an Agent's capability depends not only on the parameter count of the underlying model, but even more so on the tool ecosystem it can invoke.

Key Takeaways