Environment Duels
A dynamic agent benchmark that evaluates a large model's problem-solving and problem-posing abilities simultaneously within interactive environments. Through a privileged-hint mechanism, it quantifies the teaching and learning effects between models.
Timeline (last 90 days)
Environment Duels is a dynamic agent benchmark that evaluates a model's problem-solving and problem-setting abilities simultaneously in interactive environments.
In each round of an Environment Duels match, the model must write an interactive environment along with a privileged hint, with the goal of making a capable solver fail without the hint and pass once given it.
Evaluating problem-setting quality can be transformed into evaluating the gain of the hint, i.e., the difference in pass rate for the same solver with and without the hint.
Covering nine frontier models, the evaluation results of Environment Duels show that the strongest solver is not the strongest problem-setter, with the two capability dimensions clearly diverging.
Research shows that smaller models instead learn more from others' matches and can write the hardest environments and the most effective hints.
The quality of environments and hints has some comparable latent dimension rather than depending purely on pairing relationships.
Independent authors converge on the same designs, indicating that environment creation has a reproducible structure of "good problems."
All Facts (7)
Environment Duels is a dynamic agent benchmark that evaluates a model's problem-solving and problem-setting abilities simultaneously in interactive environments.
50%UnverifiedIn each round of an Environment Duels match, the model must write an interactive environment along with a privileged hint, with the goal of making a capable solver fail without the hint and pass once given it.
50%UnverifiedEvaluating problem-setting quality can be transformed into evaluating the gain of the hint, i.e., the difference in pass rate for the same solver with and without the hint.
50%UnverifiedCovering nine frontier models, the evaluation results of Environment Duels show that the strongest solver is not the strongest problem-setter, with the two capability dimensions clearly diverging.
50%UnverifiedResearch shows that smaller models instead learn more from others' matches and can write the hardest environments and the most effective hints.
50%UnverifiedThe quality of environments and hints has some comparable latent dimension rather than depending purely on pairing relationships.
50%UnverifiedIndependent authors converge on the same designs, indicating that environment creation has a reproducible structure of "good problems."
50%