ProteinGeoBench: Frontier Agents Excel at Structural Queries but Stumble on Geometric Computation

A Truth Masked by Scores: Good at Looking Up, Poor at Computing

In protein structure analysis, frontier AI agents display an intriguing contrast: they can quickly retrieve structural metadata but frequently fail when computing geometric relationships from coordinates. This is precisely the core issue revealed by the ProteinGeoBench benchmark—to what extent does a seemingly simple performance score reflect model capability, and to what extent does it reflect the evaluation configuration itself?

The benchmark contains 131 protein structure tasks. Researchers collected 4,249 run trajectories from four frontier models, plus 4,486 additional trials from 26 models. The results show a subtle distribution: three of the four frontier models cluster in scores between 0.785 and 0.816, differing by only 0.031 and statistically indistinguishable from one another; the fourth model scores 0.707, with differences from all three others significantly different from zero.

On the surface, this score appears to be a ranking of model capability, but the paper's authors remind us that evaluation configuration—including the number of runs, external tool access permissions, and task subset selection—may influence results no less than differences between models themselves.

Metadata Queries vs. Numerical Computation: A Divergence Between Two Task Types

A more noteworthy finding lies in the divergence across task types. Of the four frontier models, three perform better on metadata query tasks than on numerical computation tasks, with only the fourth model showing a slight advantage on numerical tasks.

The significance of this divergence lies in the weighting of task composition. Of the 131 tasks, only 28 are numerical computation tasks, yet half of the twelve "all four models fail" tasks belong to numerical computation. In other words, although numerical tasks are few, they are where failures concentrate most. When a benchmark's total score is "diluted" by a large number of metadata query tasks, a high score may mask the model's true weaknesses in geometric computation.

The researchers' recommendation is direct: evaluations should report results separately by task family rather than giving only a single total score. This means that in the future, when we see "a certain model scores 0.8 on a protein structure benchmark," we need to ask: how much of that 0.8 comes from table lookups, and how much from genuine computation?

Task family is a grouping unit commonly used in evaluation design, referring to a set of questions divided according to the cognitive or operational skills required, such as "metadata query" and "numerical computation." Questions within the same task family share similar solution procedures and failure modes, so group-level statistics can reveal structural differences masked by aggregate scores. In benchmarks, if task families have vastly different numbers of questions, the total score will be dominated by the larger family, causing a model's severe weaknesses in a minority family to be averaged away. This is precisely the methodological basis for the paper's recommendation to report results separately by task family: group reporting forces evaluators to confront the uneven distribution of capabilities rather than masking it with a single scalar.

The Fragility of Leaderboards: A Single Run May Rank Models Incorrectly

The impact of evaluation budget on rankings is equally important. The paper points out that if each task is run only once, 23.9% of bootstrap resamples would rank a different model first. This means that under a single-run evaluation setting, nearly a quarter of repeated experiments could produce a different champion.

This is not to say there are no real differences between models, but that when the magnitude of differences is on the same order as evaluation noise, the ranking itself becomes fragile. For a leaderboard used to claim "a certain model leads," this uncertainty must be disclosed.

Bootstrap resampling is a non-parametric statistical method: repeatedly drawing samples with replacement from the original evaluation results (here, with tasks as the unit), recalculating model scores and rankings each time, thereby estimating the sampling fluctuation of rankings. If the first place frequently changes hands across many resamples, the score differences between models fall within the range of evaluation noise, and the ranking is not robust. The statement "23.9% of resamples produce a different first place" means: under a single-run setting, nearly a quarter of simulated repeated experiments would yield a champion different from the original ranking. This metric directly quantifies the leaderboard's sensitivity to evaluation budget (number of runs per task).

Network Access and Tool Permissions: Variables That Are Hard to Cleanly Decompose

On a selected slice of 15 tasks, researchers compared changes across several access conditions. The score drop after removing network access was larger than the effect of blocking eight structural database domains. But the authors honestly note that these two comparisons were conducted in different evaluation rounds, making it impossible to cleanly decompose their independent effects.

Similarly, after replacing each model with its own command-line agent (harness), score changes ranged from -0.071 to +0.086. This interval is too wide to rule out moderate tool-framework effects. In other words, part of a model's score may belong to the contribution of "external scaffolding" rather than the model itself.

Harness (command-line agent framework) refers to the execution shell built around the underlying model, responsible for tool invocation, command execution, context management, and error retries. The same model connected to different harnesses may perform noticeably differently, because the harness determines whether the model can access external tools, how it parses returned results, and whether it retries after failure. Therefore, part of a benchmark score should be attributed to the harness rather than the model weights themselves. In the paper, score changes of -0.071 to +0.086 after replacing the built-in harness demonstrate that framework effects are comparable in magnitude to model effects; if evaluation reports do not disclose the harness, scores across studies cannot be directly compared.

A Cross-Sectional View of 26 Models: Numerical Ability Does Not Improve in Isolation

In extended trials with 26 models, a trend emerged: numerical computation performance rises together with non-numerical performance rather than plateauing at some level. This suggests a correlation between metadata understanding and geometric computation capability—models that look up accurately tend to also compute relatively well.

But the authors cautiously note that different experiments used different task subsets, so it is impossible to rank the four factors—model, tool framework, access conditions, and task composition—against one another. This is a methodological honesty: we know gaps exist, but we do not yet know each factor's weight.

Implications for Evaluation Practice

The ProteinGeoBench study points to a broader issue: evaluation of AI agents on scientific tasks requires more detailed reporting standards than "total-score leaderboards." The authors' recommendations include:

• Report results separately by task family rather than giving only aggregate scores
• Clearly state the number of runs per task
• Disclose the tool framework (harness) used
• Disclose network and database access conditions

For a field like structural biology that relies on precise geometric computation, "able to look up" and "able to compute correctly" are two different capabilities. An agent that can retrieve PDB entries but cannot accurately compute interatomic distances may produce seemingly reasonable but incorrect conclusions in real scientific workflows. The value of a benchmark is not in ranking models, but in revealing this structural imbalance in capabilities—which is precisely what a single score is most likely to mask.

One-line Summary

ProteinGeoBench reveals that protein structure AI agents look up well but compute poorly, with a single total score masking capability weaknesses.

Paragraph Summary

The ProteinGeoBench benchmark contains 131 protein structure tasks, with 4,249 run trajectories collected from four frontier models and 4,486 additional trials from 26 models. Results show that three frontier models cluster in scores between 0.785 and 0.816 and are statistically indistinguishable, while the fourth scores 0.707. A more critical finding is the divergence across task types: numerical computation tasks number only 28, yet account for half of all tasks failed by all models, while the large number of metadata query tasks dilutes the total score and masks geometric computation weaknesses. In addition, under single-run evaluations, 23.9% of bootstrap resamples rank a different first place, making leaderboards highly sensitive to evaluation budget; the effects of network access and tool frameworks are hard to cleanly decompose. The authors recommend reporting results separately by task family and disclosing run counts, tool frameworks, and access conditions.