KAPPO
A reinforcement learning optimization framework for LLM post-training that adapts @1 methods to @k objectives through prompt-adaptive regularization or reward transformation, thereby controlling the sharpening effect and redistributing learning pressure.
Timeline (last 90 days)
The researchers propose @k Adaptive Pressure for Policy Optimization (KAPPO), whose workflow includes a one-time reference calibration, after which existing @1 methods are adapted to @k through prompt-adaptive regularization or reward transformation.
KAPPO's experiments cover two task types, mathematical reasoning and code generation, and the results show that it provides direct control over the degree of sharpening.
Reference calibration is a one-time preprocessing step in the KAPPO workflow: before formal training, the reference model is used to estimate each prompt's baseline difficulty or baseline reward level over the task distribution, providing a normalized baseline for subsequent prompt-adaptive regularization.
All Facts (3)
KAPPO's experiments cover two task types, mathematical reasoning and code generation, and the results show that it provides direct control over the degree of sharpening.
50%UnverifiedReference calibration is a one-time preprocessing step in the KAPPO workflow: before formal training, the reference model is used to estimate each prompt's baseline difficulty or baseline reward level over the task distribution, providing a normalized baseline for subsequent prompt-adaptive regularization.
50%UnverifiedThe researchers propose @k Adaptive Pressure for Policy Optimization (KAPPO), whose workflow includes a one-time reference calibration, after which existing @1 methods are adapted to @k through prompt-adaptive regularization or reward transformation.
50%