success@k
A batch evaluation metric measuring the proportion of tasks on which a model succeeds at least once across k samples. It is commonly used to assess executability in tasks such as code generation or experimental execution, where multiple attempts are permitted.
Timeline (last 90 days)
When a coding agent performed proof-of-concept experiments on these ideas, the model's success@k was comparable to that of Claude Opus 4.8.
success@k is a batch evaluation metric: given k candidate ideas, if at least some number of them are judged successful by a downstream experiment or human, it counts as one hit; it measures whether a feasible solution can be found within a set of outputs.
All Facts (2)
When a coding agent performed proof-of-concept experiments on these ideas, the model's success@k was comparable to that of Claude Opus 4.8.
50%Unverifiedsuccess@k is a batch evaluation metric: given k candidate ideas, if at least some number of them are judged successful by a downstream experiment or human, it counts as one hit; it measures whether a feasible solution can be found within a set of outputs.
50%