AI Agents vs. Humans in First Wet-Lab Showdown: No Significant Difference in Protein Binder Design Hit Rates

On February 28, 2026, a one-day Hackathon in San Francisco quietly rewrote the history of computational biology. Nine human teams faced off against six fully autonomous LLM Agents (Claude, Quinn, Grok, Gemini, GLM, GPT), tasked with de novo designing protein Binders targeting TRAM-2, a microglial receptor associated with Alzheimer's disease. 141 designs were submitted, the Top 100 entered wet-lab validation, and ultimately 37 were confirmed as true Binders.

TRAM-2 (TREM2, Triggering Receptor Expressed on Myeloid Cells 2) is a transmembrane receptor highly expressed on the surface of brain microglia. Microglia are the resident immune cells of the central nervous system, responsible for clearing β-amyloid plaques, pruning synapses, and regulating neuroinflammation. Loss-of-function mutations in TREM2 (such as R47H) have been confirmed by genome-wide association studies (GWAS) as one of the strongest single-gene risk factors for Alzheimer's disease, increasing disease risk 2-4 fold in carriers. Designing protein Binders that bind TREM2 with high affinity could potentially activate microglial phagocytosis or serve as diagnostic probes, representing a frontier direction in neurodegenerative disease drug development.

This isn't just another benchmark leaderboard—it's the first true wet-lab closed-loop comparison between AI Agents and human protein designers. The results are both stunning and thought-provoking.

AI Agents Match Human Protein Designers: No Statistically Significant Difference in Hit Rate

Let's look at the core data: human teams achieved a Hit Rate of 38.5% (25 Binders), while Agents achieved 34.3% (12 Binders). The Fisher's exact test P-value equals 0.83, and the Mann-Whitney test P-value for pKD distribution equals 0.75—statistically speaking, there is no significant difference between the two.

It's worth explaining what these two statistical tests mean. Fisher's exact test is a non-parametric statistical method for analyzing 2×2 contingency tables, particularly suitable for small sample sizes—it compares whether the Binder success/failure ratios between human and Agent groups show significant differences. A P-value of 0.83 means there's an 83% probability that the observed difference is due to random fluctuation alone, far above the typical significance threshold of 0.05. The Mann-Whitney U test compares the median difference in pKD distributions between the two groups, and a P-value of 0.75 similarly indicates no statistically significant difference in affinity distributions. Together, these two tests form the rigorous statistical foundation for the conclusion that "AI has caught up with humans."

The top three on the leaderboard were indeed all human, but the Agent's best result ranked fourth, with the strongest human only about three-fold better than the strongest Agent. Considering that each Agent received only a single Brief and was then dropped into Muni's Sandbox—an isolated computing environment designed for LLM Agents where they can access pre-installed protein design toolchains, execute computational tasks, read output results, and make subsequent decisions—choosing their own tools, running their own Pipeline, ranking results, and submitting their Final 10, this performance is remarkable. This end-to-end process encompasses four core aspects of research: problem understanding, solution design, experiment execution, and result evaluation, making it one of the most complete end-to-end capability tests for AI Agents in scientific discovery to date.

Even more noteworthy is the historical longitudinal comparison. The first Adaptive competition targeting EGFR had a Hit Rate of only 2.5%, the second round improved to 13.2%, and January's NIPA task was 9.6%. This TRAM-2 task's overall Hit Rate broke through 30%, setting a new high-water mark for Binder design in the Post-AlphaFold era. The Adaptive team calls it "Moore's Law for Binder Hit Rates."

To understand the significance of this progress, one needs to understand the basic principles of protein Binder design. Designing a completely novel protein molecule from scratch to bind a target protein with high affinity requires precise calculation of shape complementarity, hydrogen bond networks, hydrophobic packing, and electrostatic matching at the protein-protein interaction interface. Traditional methods relied on energy function-driven computational frameworks like Rosetta, while the emergence of AlphaFold2 dramatically improved structural prediction accuracy, spawning a new generation of deep learning-based design tools such as RFdiffusion and BindCraft. The core metrics for evaluating Binder success are precisely Hit Rate (the proportion of submitted designs that actually bind the target) and pKD (negative logarithm of the dissociation constant—higher values indicate stronger affinity). The leap from 2.5% to 30% Hit Rate marks these new tools transitioning from "occasionally successful" to "reliably productive."

In-Silico Scorer Lacks Precision: Agent Designs Disproportionately Eliminated

However, the "matching" conclusion is built upon an imprecise screening mechanism.

How were 141 designs filtered to the Top 100? Using BoSe 2's iPSAE scoring. iPSAE (integrated Protein Structure and Energy) is an in-silico scoring function that comprehensively considers structural prediction confidence, interface energy, and contact surface area features. But when the Muni team retrospectively analyzed after completing wet-lab experiments, they found that iPSAE's binary classification prediction for "whether something is a Binder" had an AUC of only 0.65—somewhat better than random guessing, but far from a reliable classifier.

AUC (Area Under the ROC Curve) is the standard metric for evaluating binary classifier performance: AUC=1.0 indicates perfect classification, AUC=0.5 is equivalent to random guessing. iPSAE's AUC of only 0.65 means its accuracy in distinguishing "true Binders" from "non-binding designs" is only marginally better than flipping a coin. This finding carries warning significance for the entire computational biology field—many studies rely on similar in-silico metrics to screen candidate molecules, but the actual predictive power of these metrics may be far lower than expected.

More subtly, this inaccurate Filter cut Agent designs more aggressively: Agents lost 42% of their designs, while humans lost only 20%. In other words, with a more accurate screening method, more Agent designs might have entered wet-lab validation, and the final Hit Rate could potentially have been higher.

This reveals a deeper problem: The precision of current in-silico evaluation tools has become the core bottleneck constraining progress in protein design. An AUC of 0.65 means our ability to predict design quality remains weak—only wet-lab closed-loop validation can reveal the truth.

Six LLM Agents All Choose PX Design: Tool Convergence Is Concerning

This is the most spine-chilling finding in the entire report.

Muni's tool menu was extensive: PX Design, BindCraft, RF Diffusion, BoSe Gen, and more. These tools represent fundamentally different technical approaches—PX Design is based on a hybrid architecture of physical energy functions and deep learning; RFdiffusion, developed by David Baker's lab, uses diffusion models to gradually generate protein backbone structures from noise, making it one of the most influential de novo design methods currently available; BindCraft focuses on leveraging AlphaFold2's structural prediction capabilities to inversely optimize sequences, iteratively refining them until the predicted structure converges on an ideal binding configuration. Different tools represent different design philosophies—from physics-driven to data-driven, from global generation to modular assembly.

Yet six Agents from completely different companies, with no communication between them, all independently chose PX Design. Claude, Quinn, Grok, Gemini, GLM, GPT—all six.

PX Design accounted for 53% of Agent-submitted designs, and among the Agents' Top 10 Binders, all 10 came from PX Design.

Why PX Design? It does perform well—it published a Preprint just last August with excellent marketing, claiming 17-82% Hit Rates. Agents may have discovered it through training data after the Knowledge Cutoff, tool list descriptions, or Web Search. The specific mechanism doesn't matter; what matters is the result: When AI Agents become the entry point for research Pipelines, whoever's tool gets "seen" and preferentially selected by Agents wins.

This means an entirely new competitive dimension is forming—tool developers must not only gain recognition from human scientists but also ensure AI Agents can discover and preferentially select their tools. This is essentially the research version of "SEO for AI Agents," and a completely new challenge facing the computational biology tool ecosystem.

Brute Force Generation vs. Expert Judgment: The Fundamental Difference Between Humans and AI Agents

Breaking down by tool reveals even starker human-machine differences.

The champion team NRes used the Mosaic method—a design approach employing modular assembly strategies that recombine known effective structural modules to construct new binding interfaces—achieving 4 successes out of 6 designs, a Hit Rate of 66.7%. Notably, Mosaic wasn't even on Muni's official tool menu, meaning the champion team proactively introduced a non-standard tool choice, demonstrating human designers' creative judgment at the methodological level. Using the same PX Design tool, humans achieved a 72.7% Hit Rate, far exceeding the Agents' 44.4%.

But interestingly, the Agents that "maxed out" PX Design actually produced the highest-affinity Binder—3.64 nanomolar. This number ranks near the top across the entire competition. In the drug development context, lower KD values indicate tighter Binder-target binding. Generally, molecules with KD at the micromolar (μM) level are considered preliminary hits, nanomolar-level enters lead compound territory, and clinical candidates typically need to reach sub-nanomolar or even picomolar levels. A 3.64 nM affinity means this design already has potential as a lead molecule for further optimization, potentially entering functional validation without extensive affinity maturation steps. For a protein designed from scratch in a single-day Hackathon, this number is particularly impressive.

This reveals the fundamental difference between humans and AI Agents in protein design: Agents rely on brute-force mass generation, while humans rely on judgment for precise design. Human designers choose different tools based on experience, adjust parameters, and make non-standardized judgments; Agents tend to find the "best-looking" tool and mass-generate, trading quantity for quality.

Both strategies have their merits, but as computational costs continue to decline, the cost-effectiveness of brute-force strategies will only improve.

The Future of Protein Designers: Not Whether They'll Be Replaced, But How to Transform

Returning to the core question: In the AI Agent era, are protein designers' jobs safe?

In the short term, the real danger isn't for senior designers, but for those who can only run a single Pipeline. The true signal from this Hackathon is that three things happened simultaneously:

1. AI Agent end-to-end capabilities have matched human Hit Rates—from understanding tasks, selecting tools, running computations to submitting results, all without human intervention
2. Existing in-silico Filters are severely inadequate (AUC only 0.65)—wet-lab closed-loop validation remains irreplaceable
3. Agents show severe homogeneity in tool selection—lacking the diversity of thought and non-consensus judgment of human designers

Designers' jobs are safe, but the way they hold those jobs must change. Future protein designers need three core capabilities: the ability to understand and guide AI Agent behavior patterns, the ability to make non-consensus judgments in tool selection, and the ability to design and interpret wet-lab experiments.

When AI Agents can complete the entire process from Brief to Final Design in a single day, designers' value is no longer in "running Pipelines," but in defining problems, choosing non-obvious paths, and making final judgments on Agent outputs. This one-day Hackathon in San Francisco isn't an endpoint—it's the starting point of a new era of human-machine collaborative protein design.

Key Takeaways

• Six fully autonomous LLM Agents matched humans for the first time in protein Binder design wet-lab experiments, with Hit Rates of 34.3% and 38.5% respectively—no statistically significant difference
• The existing in-silico screening tool (iPSAE) has an AUC of only 0.65, with a much higher elimination rate for Agent designs (42%) than human designs (20%)—wet-lab closed-loop validation remains the only path to truth
• Six Agents from different companies all independently chose PX Design without communication, exposing severe tool selection convergence
• Human designers achieved higher per-tool Hit Rates through judgment and tool diversity, but Agents produced the highest-affinity Binder (3.64 nanomolar) through brute-force strategies
• The core value of protein designers is shifting from Pipeline execution to problem definition, non-consensus tool selection, and wet-lab judgment