Agent Deployment Stuck at Evaluation: Snorkel AI Breaks Down Six Rules for Effective Benchmarks

Why Is Enterprise Adoption Lagging Despite Agent Capability Gains?

Snorkel AI co-founder and research scientist Vincent raised an industry observation during a technical talk: Agent capability curves are continually climbing, and improvements are especially noticeable in coding scenarios. But when he spoke with enterprises in high-stakes industries like finance, insurance, and healthcare, they generally expressed hesitation. The problem isn't model capability itself—it's that our ability to measure how Agents actually perform lags behind the pace of capability improvement.

This "evaluation gap" is becoming a real bottleneck for Agent deployment. Snorkel positions itself as a frontier AI data lab, with co-founders holding labs at Stanford, the University of Washington, and the University of Wisconsin, along with a deployment engineer team serving enterprises. This dual exposure to academic frontiers and real-world enterprise practice gives them firsthand insight into evaluation problems.

Vincent revealed that Snorkel has invested $3 million in an open benchmark grant program and has reviewed over 120 applications from academia and industrial labs. He attempted to distill two methodologies from this practice: the science side—how to build effective measuring sticks; and the art side—what kinds of benchmarks can truly shape the direction of a field.

The Science Side: Four Foundational Elements of Effective Benchmarks

Task Quality: Not Just "Writing Questions," but Rigorous Adversarial Validation

Individual task quality is the foundation of a benchmark. Vincent emphasized three criteria: tasks must represent real-world complexity, instructions must be well-structured, and answers must be verifiable and confirmed by domain experts. He used GPQA as a model example—a benchmark capturing graduate and professional-level knowledge that still appears on model cards today. But Vincent believes its most underappreciated contribution is hidden in the appendix: a set of adversarial quality control mechanisms.

Specifically, it introduces a multi-role protocol involving original authors, reviewers, and adjudicators, allows revisions, and designs an incentivized compensation mechanism based on expert consensus. This process borrows from academic peer review thinking, solving the challenge of frontier knowledge tasks where "even a single expert struggles to judge quality."

GPQA (Google-Proof Q&A) is an expert-level question-answering benchmark released in 2023. Questions are written by PhD-level experts in physics, biology, and chemistry, with the design goal that they are "difficult to answer even with search engines." It contains approximately 448 multiple-choice questions, each accompanied by detailed expert-written explanations. GPQA-Diamond is a high-difficulty subset after quality screening, often cited in frontier model reports. The benchmark's unique value lies in demonstrating how to use multi-round expert review and consensus voting to ensure question quality—each question undergoes independent verification and cross-challenge by at least three domain experts, which is why Vincent regards it as a model of "adversarial quality control."

MMLU (Massive Multitask Language Understanding) was introduced in 2020 and contains 57 subjects with about 15,000 multiple-choice questions covering STEM, humanities, social sciences, and professional domains (such as law and medicine). It was the first benchmark to systematically measure model "breadth of knowledge" and has long served as a general yardstick for model capability. Its 57-subject taxonomy became a reference template for subsequent benchmark design, which is empirical evidence for Vincent's emphasis that "taxonomy is the source of a benchmark's vitality." However, as frontier models approach saturation on MMLU, its discriminative power has significantly declined.

Distribution Diversity: Using Taxonomy to Combat "Long-Tail Imbalance"

Meaningful benchmarks need to define clear taxonomies for the domain and consciously allocate tasks. Vincent gave the classic autonomous driving example: scenarios like yellow lights, pedestrians, and motorcyclists appear far less frequently in real data than their importance warrants—they are "rare but critical" failure modes. Benchmark designers must actively characterize and cover these long tails.

MMLU's taxonomy of 57 subjects and professional domains is considered one source of its long-term vitality.

Difficulty and Model Headroom: Benchmarks Must Be "Non-Saturated"

Good benchmarks should continuously expose model weaknesses and stably distinguish frontier models' positions. ARC-AGI is Vincent's favorite case: ARC-AGI 2 remained unsaturated for a long time, and the performance jumps during the reasoning capability advancement period were highly correlated with real capability leaps. When ARC-AGI 3 was released, frontier models scored below 1% on all tasks, while every task was solvable by humans—this "humans can do it, models can't" headroom design makes it one of the most anticipated evaluations at model releases.

ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) was proposed by François Chollet in 2019 and contains a series of grid transformation puzzles requiring abstract rule induction from few examples. Its design philosophy is "every task is solvable by humans but difficult for machine learning"—humans typically complete them in minutes, while systems need compositional generalization capability. ARC-AGI 2 remained unsaturated for a long time before reasoning models (like the o-series) emerged; ARC-AGI 3 introduces interactive environments to further test dynamic planning and exploration capabilities.

"Saturation" refers to the state where frontier models' scores on a benchmark approach or reach the human ceiling, causing the benchmark to lose discriminative power. A saturated benchmark can no longer expose model weaknesses or provide meaningful signals for model releases. Therefore, healthy benchmarks should deliberately preserve capability headroom where "humans can do it, models can't," and through tiered difficulty design, allow stable discrimination across model generations.

Evaluation Methodology: Real Dimensions Beyond Accuracy

Robust evaluation must go beyond accuracy to capture real dimensions like cost, latency, reasoning trajectory quality, and tool use, treating these signals as rewards or supervision signals. Vincent used tau-bench as an example: it evaluates multi-turn Agent task completion, designs a clever user simulator, and measures policy constraint compliance—a model that books the right flight but violates cabin class rules is still judged as a failure.

tau-bench (Tool-Agent-User Benchmark) is a multi-turn Agent evaluation benchmark proposed by Sierra AI, focusing on tool calling and user interaction scenarios (such as airline booking and retail customer service). Its core design is a user simulator: an LLM plays the user in multi-turn conversations, letting the Agent complete tasks through dynamic interaction. Evaluation not only checks task completion but also judges through policy compliance—for example, if the Agent books the right flight but violates cabin class rules, it is judged as a failure. This embodies Vincent's emphasis on "multi-dimensional signals": Agent success is not just about correct results but also process compliance, appropriate tool use, and natural interaction.

The Art Side: Three Differentiating Characteristics of Great Benchmarks

Having a Thesis: Betting on the Direction of the Field

The most ambitious benchmarks are essentially declarations about where the world is heading. Terminal-bench bets that CLI is not only the interface for coding Agents but also the core abstraction for general computer use. Judging from how teams like Claude and Codex are currently building enterprise-grade general capabilities on top of CLI, this early bet has proven quite correct and influential.

Paving the Way: Spawning New Research Directions

Good benchmarks produce new roadmaps and inspire new research attack directions. SWE-bench used a simple idea—leveraging PRs to reuse existing coding capability—to spawn a family including SWE-bench Lite and Verified, continuously influencing how coding Agents are conceived, with substantial innovation space remaining, such as how new abstraction layers like vibe coding should be evaluated.

Valuing Researcher Experience: The Underappreciated Adoption Lever

Vincent considers this a "severely underappreciated" dimension. The sharpest benchmark builders invest effort in making models and Agents easy to run on the benchmark, making it easy for the community to contribute new tasks, and making benchmark signals reusable for RL or post-training. Stanford CRFM's HELM pioneered standardized modular evaluation frameworks, while Terminal-bench 2.0 released Harbor—a de facto Agent evaluation infrastructure.

SWE-bench is a software engineering benchmark proposed in 2023, where models generate patches based on issue descriptions in real GitHub repositories and validate them through tests. Its key innovation is reusing existing coding capability: building tasks directly from open-source projects' PRs and test cases, greatly reducing annotation costs. It subsequently spawned a family including SWE-bench Lite (a streamlined subset) and SWE-bench Verified (a manually verified high-quality subset), becoming the de facto standard for coding Agent evaluation and continuously spawning new research directions.

HELM (Holistic Evaluation of Language Models) is a standardized evaluation framework initiated by Stanford CRFM, emphasizing modularity, reproducibility, and multi-dimensional reporting (accuracy, robustness, fairness, efficiency, etc.). It was the first to treat "evaluation infrastructure" itself as a research object, driving the shift from one-off scoring to sustainable engineering platforms—precisely the pioneering example of what Vincent calls "researcher experience as an adoption lever."

Three New Directions Snorkel Is Bullish On

Vincent offered three axes for next-generation benchmarks:

Environment complexity. Real codebases contain organization-specific policies, Slack context, screenshots, scattered CI, human reviewers with implicit preferences, and parallel contributors—current benchmarks capture only a small fraction. This is where Agents fail most frequently today.

Autonomy duration. How long can an Agent operate before reliability collapses? Take customer service Agents: context passed weeks ago may be lost, product specs may change, and organizational restructuring may shift priorities mid-course. State changes under long-horizon, continually learning settings require new benchmarks to characterize.

Output complexity and signal granularity. For outputs like strategic advice and complex reports, "good" is subjective and hard to verify, requiring capture of organizational context and human judgment. Vincent specifically mentioned interest in "trustworthy outputs"—whether Agents can express their own uncertainty, proactively stop or request more information, and interaction forms beyond plain text.

Structural Signals Worth Noting

The value of this talk goes beyond a checklist. It reveals an often-overlooked causal chain: benchmarks are not just snapshots looking back at progress—they are coordinates that define progress and set capability goals. When a field's capability grows faster than its measurement capability, lagging commercial adoption is almost inevitable—enterprises don't distrust models; they simply cannot prove models are reliable in their specific scenarios.

For builders, this means evaluation capability is upgrading from "incidental work" to core competitiveness. For researchers, Snorkel is still soliciting projects in its open benchmark grant program, with public email benchmarks.snorkel.ai—perhaps one of the few channels currently providing direct funding for benchmark innovation.

One-line Summary

Agent capability grows faster than evaluation capability, and the evaluation gap is becoming the core obstacle to enterprise adoption.

Paragraph Summary

Snorkel AI co-founder Vincent points out that Agent capability continues to improve, yet enterprises in high-stakes industries like finance, insurance, and healthcare remain hesitant. The core reason is not insufficient model capability but that the ability to measure how Agents actually perform lags behind the pace of capability improvement, creating an "evaluation gap." He summarizes four elements of effective benchmarks on the science side: rigorous adversarial task validation (exemplified by GPQA), taxonomy-driven distribution control (combating long-tail imbalance), maintaining unsaturated model headroom (like ARC-AGI), and multi-dimensional evaluation beyond accuracy (like tau-bench). From the art side, he distills three characteristics of great benchmarks: betting on the field's direction (Terminal-bench betting on CLI), spawning new research directions (the SWE-bench family), and valuing researcher experience (HELM, Harbor). Finally, he offers three directions Snorkel is bullish on for next-generation benchmarks: environment complexity, autonomy duration, and output complexity and uncertainty expression. Snorkel has invested $3 million to fund open benchmarks and reviewed over 120 applications.