Pass@k
A standardized metric for evaluating the code generation capability of language models, introduced by OpenAI in the HumanEval benchmark. Pass@k measures the probability that at least one correct solution is generated among k sampled attempts, and has been widely adopted in the evaluation of Agent systems.
Timeline (last 90 days)
Pass@k is a standardized metric for evaluating the code generation capability of language models, introduced by OpenAI in the HumanEval benchmark.
Pass@1 represents the probability that the model generates a correct result in a single sampling, while Pass@k represents the probability of at least one correct result in k samplings.
All Facts (2)
Pass@k is a standardized metric for evaluating the code generation capability of language models, introduced by OpenAI in the HumanEval benchmark.
50%UnverifiedPass@1 represents the probability that the model generates a correct result in a single sampling, while Pass@k represents the probability of at least one correct result in k samplings.
50%