pass@1
A standard scoring convention in code generation evaluation in which the model produces a single answer per task; a passing result scores 1 and a failing result scores 0, and the final score is the average across all tasks, measuring single-attempt delivery success rate.
Timeline (last 90 days)
pass@1 measures the proportion of first responses that conform to the schema, pass@repair measures the cumulative usable proportion after one repair, and reject rate measures the proportion that should be rejected.
After GRPO reinforcement learning, checkpoints using TailSFT achieved a final pass@1 improvement of up to 3.9 points.
If pass@16 is much higher than pass@1, it indicates that the model's probability distribution contains the correct answer but does not place it at the highest probability position, which is precisely the space where RL can exert its effect through reward signals.
All Facts (3)
pass@1 measures the proportion of first responses that conform to the schema, pass@repair measures the cumulative usable proportion after one repair, and reject rate measures the proportion that should be rejected.
50%UnverifiedAfter GRPO reinforcement learning, checkpoints using TailSFT achieved a final pass@1 improvement of up to 3.9 points.
50%UnverifiedIf pass@16 is much higher than pass@1, it indicates that the model's probability distribution contains the correct answer but does not place it at the highest probability position, which is precisely the space where RL can exert its effect through reward signals.
50%