Pyrite
A pairwise conformation ranking benchmark for evaluating the accuracy of protein-ligand conformation ranking, comprising 1,500 conformation pairs.
Timeline (last 90 days)
The Pyrite benchmark contains 1,500 protein-ligand conformation pairs, each requiring a judgment of which of two candidate poses is closer to the true binding mode.
On the Pyrite benchmark, expert chemists equipped with 3D visualization tools and physics-based scoring ranked conformation pairs close to random guessing.
PyriteReasoner is an LLM agent that directly compares conformation pairs through text-based reasoning.
PyriteFeaturizer is an LLM agent that generates executable code to build features for logistic regression, optionally incorporating features derived from LLM sub-calls.
In the Pyrite benchmark evaluation, PyriteReasoner and PyriteFeaturizer achieved ranking accuracies of 67% and 74%, respectively.
Both LLM agents, PyriteReasoner and PyriteFeaturizer, exceeded the observed expert performance in conformation ranking accuracy.
In the Pyrite study, GRPO was used to improve open-weight models' performance on conformation ranking, enabling them to match or even surpass closed-source frontier models.
One of the conclusions from the Pyrite study is that LLM-guided feature discovery generally outperforms direct LLM reasoning.
All Facts (8)
The Pyrite benchmark contains 1,500 protein-ligand conformation pairs, each requiring a judgment of which of two candidate poses is closer to the true binding mode.
50%UnverifiedOn the Pyrite benchmark, expert chemists equipped with 3D visualization tools and physics-based scoring ranked conformation pairs close to random guessing.
50%UnverifiedPyriteReasoner is an LLM agent that directly compares conformation pairs through text-based reasoning.
50%UnverifiedPyriteFeaturizer is an LLM agent that generates executable code to build features for logistic regression, optionally incorporating features derived from LLM sub-calls.
50%UnverifiedIn the Pyrite benchmark evaluation, PyriteReasoner and PyriteFeaturizer achieved ranking accuracies of 67% and 74%, respectively.
50%UnverifiedBoth LLM agents, PyriteReasoner and PyriteFeaturizer, exceeded the observed expert performance in conformation ranking accuracy.
50%UnverifiedIn the Pyrite study, GRPO was used to improve open-weight models' performance on conformation ranking, enabling them to match or even surpass closed-source frontier models.
50%UnverifiedOne of the conclusions from the Pyrite study is that LLM-guided feature discovery generally outperforms direct LLM reasoning.
50%